Information processing apparatus, information processing method, and program

ABSTRACT

There is provided an information processing apparatus, an information processing method, and a program that provide a mechanism with which the display location of the operation object is able to be operated with a sensation of displacing an actual object. The information processing apparatus includes: a display control unit that controls display of an operation object for a device to be operated; and a reference control unit that controls a reference of a location at which the operation object is displayed such that the operation object is able to be visually recognized, on the basis of a predetermined operation performed to the operation object by an operation subject of the device to be operated.

TECHNICAL FIELD

The present disclosure relates to an information processing apparatus,an information processing method, and a program.

BACKGROUND ART

In recent years, a variety of interfaces for controlling devices havebeen researched and developed with development of informationcommunication technologies. As one of such interfaces, there is aninterface using a displayed virtual object.

For example, an invention in which an information processing apparatusexecutes processing in accordance with a user's motion relative to aprojected image has been disclosed in Patent Literature 1, for example.Further, causing the projected image to follow a user's hand motion whena motion of gripping the projected image is detected is disclosed as oneof interactions between such an information processing apparatus and theuser in Patent Literature 1 (FIG. 44 and the like).

CITATION LIST Patent Literature

Patent Literature 1: JP 2011-188023A

DISCLOSURE OF INVENTION Technical Problem

However, it is desired to control a display location of an operationobject, such as a projected image as described above, with an operationthat is closer to an operation performed to an actual object. Thus, thepresent disclosure proposes a mechanism with which the display locationof the operation object is able to be operated with a sensation ofdisplacing an actual object.

Solution to Problem

According to the present disclosure, there is provided an informationprocessing apparatus including: a display control unit that controlsdisplay of an operation object for a device to be operated; and areference control unit that controls a reference of a location at whichthe operation object is displayed such that the operation object is ableto be visually recognized, on the basis of a predetermined operationperformed to the operation object by an operation subject of the deviceto be operated.

In addition, according to the present disclosure, there is provided aninformation processing method including, using a processor: controllingdisplay of an operation object for a device to be operated; andcontrolling a reference of a location at which the operation object isdisplayed such that the operation object is able to be visuallyrecognized, on the basis of a predetermined operation performed to theoperation object by an operation subject of the device to be operated.

In addition, according to the present disclosure, there is provided aprogram for causing a computer system to achieve: a display controlfunction of controlling display of an operation object for a device tobe operated; and a reference control function of controlling a referenceof a location at which the operation object is displayed such that theoperation object is able to be visually recognized, on the basis of apredetermined operation performed to the operation object by anoperation subject of the device to be operated.

Advantageous Effects of Invention

According to the present disclosure, a mechanism with which a displaylocation of an operation object is able to be operated with a sensationof displacing an actual object is provided. Note that the effectsdescribed above are not necessarily limitative. With or in the place ofthe above effects, there may be achieved any one of the effectsdescribed in this specification or other effects that may be graspedfrom this specification.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for describing an outline of an informationprocessing system according to one embodiment of the present disclosure.

FIG. 2 is a flowchart illustrating an outline of processing of theinformation processing system according to one embodiment of the presentdisclosure.

FIG. 3 is a block diagram schematically illustrating an example of afunctional configuration of an information processing system accordingto a first embodiment of the present disclosure.

FIG. 4 is a diagram for describing an example of first device selectionthrough which one device to be operated is selected in the informationprocessing system according to the embodiment.

FIG. 5 is a diagram for describing an example of first device selectionthrough which a plurality of devices to be operated are selected in theinformation processing system according to the embodiment.

FIG. 6 is a diagram for describing an example for describing an exampleof second device selection through which an operation target device isselected on the basis of an operation performed to the selection objectin the information processing system according to the embodiment.

FIG. 7 is a diagram for describing an example of priority information inthe information processing system according to the embodiment.

FIG. 8 is a diagram illustrating an example of a notification of devicesto be operated, which have been selected through first device selectionin the information processing system according to the embodiment.

FIG. 9 is a flowchart conceptually illustrating an example of overallprocessing of the information processing system according to theembodiment.

FIG. 10 is a flowchart conceptually illustrating an example of selectionobject display processing of the information processing system accordingto the embodiment.

FIG. 11 is a diagram for describing an example of first device selectionin an information processing system according to a first modificationexample of the embodiment.

FIG. 12 is a diagram for describing another example of the first deviceselection in the information processing system according to the firstmodification example of the embodiment.

FIG. 13 is a diagram for describing yet another example of the firstdevice selection in the information processing system according to thefirst modification example of the embodiment.

FIG. 14 is a flowchart conceptually illustrating an example of selectionobject display processing of an information processing system accordingto a second modification example of the embodiment.

FIG. 15 is a diagram illustrating an example of display of selectionobjects in an information processing system according to a thirdmodification example of the embodiment.

FIG. 16 is a diagram illustrating another example of display of theselection objects in the information processing system according to thethird modification example of the embodiment.

FIG. 17 is a flowchart conceptually illustrating an example of selectionobject display processing of an information processing system accordingto a fourth modification example of the embodiment.

FIG. 18 is a flowchart conceptually illustrating another example ofselection object display processing of the information processing systemaccording to the fourth modification example of the embodiment.

FIG. 19 is a diagram illustrating an example of a notification ofdevices to be operated, which have been selected through first deviceselection in an information processing system according to a fifthmodification example of the embodiment.

FIG. 20 is a diagram illustrating another example of a notification ofdevices to be operated, which have been selected through the firstdevice selection in the information processing system according to thefifth modification example of the embodiment.

FIG. 21 a diagram illustrating yet another example of a notification ofdevices to be operated, which have been selected through the firstdevice selection in the information processing system according to thefifth modification example of the embodiment.

FIG. 22 is a diagram illustrating an example of display control foroperation objects in an information processing system according to asecond embodiment of the present disclosure.

FIG. 23 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto the embodiment.

FIG. 24 is a flowchart conceptually illustrating an example ofprocessing of the information processing system according to theembodiment.

FIG. 25 is a diagram illustrating an example of display control foroperation objects in an information processing system according to afirst modification example of the embodiment.

FIG. 26 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto the first modification example of the embodiment.

FIG. 27 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto a second modification example of the embodiment.

FIG. 28 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto the second modification example of the embodiment.

FIG. 29 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto a third modification example of the embodiment.

FIG. 30 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto the third modification example of the embodiment.

FIG. 31 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto a fourth modification example of the embodiment.

FIG. 32 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto a fifth modification example of the embodiment.

FIG. 33 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto a sixth modification example of the embodiment.

FIG. 34 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto the sixth modification example of the embodiment.

FIG. 35 is a diagram illustrating another example of the display controlfor the operation objects in the information processing system accordingto the sixth modification example of the embodiment.

FIG. 36 is a flowchart conceptually illustrating an example ofprocessing of an information processing system according to a sixthmodification example of the embodiment.

FIG. 37 is a diagram illustrating an example of display control fornotification operation objects in an information processing systemaccording to a seventh modification example of the embodiment.

FIG. 38 is a diagram illustrating an example of display control fornotification operation objects for a plurality of users in theinformation processing system according to the seventh modificationexample of the embodiment.

FIG. 39 is a diagram illustrating an example of display control fornotification operation objects, which are visually recognized only by aspecific user, in the information processing system according to theseventh modification example of the embodiment.

FIG. 40 is a diagram illustrating another example of the display controlof the notification operation objects, which are visually recognizedonly by the specific user, in the information processing systemaccording to the seventh modification example of the embodiment.

FIG. 41 is a diagram illustrating an example of display control foroperation objects in an information processing system according to aneighth modification example of the embodiment.

FIG. 42 is a diagram illustrating an example of display control foroperation objects in an information processing system according to aninth modification example of the embodiment.

FIG. 43 is a diagram illustrating an example of an operation performedto an operation object in an information processing system according toa tenth modification example of the embodiment.

FIG. 44 is a diagram illustrating an example of displacement control ofoperation objects in an information processing system according to athird embodiment of the present disclosure.

FIG. 45 is a diagram illustrating an example of followability of theoperation objects relative to a reference of a display location in theinformation processing system according to the embodiment.

FIG. 46 is a diagram illustrating an example of followability of theoperation objects in a case in which the reference of the displaylocation is a person in the information processing system according tothe embodiment.

FIG. 47 is a flowchart conceptually illustrating an example of overallprocessing of the information processing system according to theembodiment.

FIG. 48 is a flowchart conceptually illustrating an example ofprocessing of controlling a reference of the display location in theinformation processing system according to the embodiment.

FIG. 49 is a diagram illustrating an example of a predeterminedoperation in an information processing system according to a firstmodification example of the embodiment.

FIG. 50 is a diagram illustrating another example of a predeterminedoperation in the information processing system according to the firstmodification example of the embodiment.

FIG. 51 is a diagram illustrating yet another example of a predeterminedoperation in the information processing system according to the firstmodification example of the embodiment.

FIG. 52 is a diagram illustrating yet another example of a predeterminedoperation in the information processing system according to the firstmodification example of the embodiment.

FIG. 53 is a diagram illustrating yet another example of a predeterminedoperation in the information processing system according to the firstmodification example of the embodiment.

FIG. 54 is a diagram illustrating an example in which a reference of adisplay location is changed in an information processing systemaccording to a second modification example of the embodiment.

FIG. 55 is a flowchart conceptually illustrating an example ofprocessing of controlling a reference of a display location in aninformation processing system according to a third modification exampleof the embodiment.

FIG. 56 is a diagram illustrating an example in which a destinationafter changing of a reference of a display location of an operationobject is explicitly indicated in an information processing systemaccording to a fourth modification example of the embodiment.

FIG. 57 is a diagram illustrating an example of merging of operationobjects in an information processing system according to a fifthmodification example of the embodiment.

FIG. 58 is a diagram illustrating an example of display control foroperation objects in an information processing system according to asixth modification example of the embodiment.

FIG. 59 is a diagram illustrating an example of display control foroperation objects in an information processing system according to aseventh modification example of the embodiment.

FIG. 60 is a diagram illustrating an example of display control foroperation objects related to reproduction in an information processingsystem according to a seventh modification example of the embodiment.

FIG. 61 is an explanatory diagram illustrating a hardware configurationof an information processing apparatus according to one embodiment ofthe present disclosure.

DESCRIPTION OF EMBODIMENTS

Hereinafter. (a) preferred embodiment(s) of the present disclosure willbe described in detail with reference to the appended drawings. Notethat, in this specification and the appended drawings, structuralelements that have substantially the same function and structure aredenoted with the same reference numerals, and repeated explanation ofthese structural elements is omitted.

In addition, different numbers may be added to the same referencenumbers for distinguishing a plurality of elements that havesubstantially the same functions in the specification and drawings insome cases. For example, a plurality of elements that have substantiallythe same functions are distinguished like a selection object 31A, aselection object 31B, and the like as needed. However, only the samereference numerals are given in a case in which it is not necessary todistinguish elements that have substantially the same functions. In acase in which it is not necessary to particularly distinguish theselection object 31A and the selection object 31B, for example, theselection object 31A and the subject object 31B will be simply referredto as selection objects 31.

In addition, information processing apparatuses 100 according to firstto third embodiments will be distinguished as an information processingapparatus 100-1 to an information processing apparatus 100-3 by addingthe numbers corresponding to the embodiments to the end for convenienceof description.

Note that the description will be given in the following order.

1. Introduction

2. Outline of information processing system according to one embodimentof the present disclosure3. First embodiment of the present disclosure (selection of device to beoperated)3-1. Configuration of system3-2. Processing of system3-3. Summary of first embodiment3-4. Modification examples4. Second embodiment of the present disclosure (display of operationobjects)4-1. Configuration of system4-2. Processing of system4-3. Summary of second embodiment4-4. Modification examples5. Third embodiment of the present disclosure (displacement of operationobjects)5-1. Configuration of system5-2. Processing of system5-3. Summary of third embodiment5-4. Modification examples6. Hardware configuration according to one embodiment of the presentdisclosure

7. Conclusion 1. Introduction

First, technologies related to the information processing systemsaccording to the respective embodiments of the present disclosure willbe described.

Electronic devices (hereinafter. also referred to as devices to beoperated) such as domestic electrical appliances are generally operatedby using remote controllers. For example, remote controllers areprovided for the respective devices to be operated, and a user operatesa device to be operated, which is an operation target, by using a remotecontroller for the device to be operated.

However, since the number of remote controllers also increases as thenumber of devices to be operated increases, there is a concern that itmay become complicated for the user to operate the devices to beoperated. For example, user's efforts to find a remote controller for adesired device to be operated increase. Also, management of the remotecontrollers also becomes an additional burden.

Meanwhile, operating a plurality of devices to be operated by using onedevice corresponding to the remote controllers is conceivable. Forexample, switching GUIs (Graphical User Interfaces) for operating thedevices to be operated in accordance with the devices to be operated ina device in which a GUI is displayed may be exemplified.

However, complication still remains for the user in this case. Forexample, the user has to pick up the aforementioned device with his/herhand to operate the devices to be operated. Also, since a desired GUI isselected from among the GUIs for the plurality of devices to beoperated, the number of GUIs also increases as the number of devices tobe operated increases, and selection of the GUIs becomes complicated. Inaddition, it thus takes time until it becomes possible to perform anoperation. Further, in a case in which there are a plurality of users,it is necessary to provide the aforementioned devices corresponding tothe number of users.

In consideration of the above points, providing an operation interfacein which the aforementioned devices are not used is conceivable. Forexample, devices to be operated recognizing a so-called NUI (NaturalUser Interface) such as speech, lines of sight, or gestures of the userand executing an operation that the user desires may be exemplified.

However, in a case in which a plurality of devices to be operated arepresent, it is generally difficult to select a desired device to beoperated through an NUI. For example, if a user tries to select a deviceto be operated by using a line of sight, a line sight not only to thedesired device to be operated but also to devices to be operated in thesurroundings may be recognized due to deviation of the line sight, andimproper operating may occur. Also, curbing of deviation in a line ofsight is a burden on the user. Further, in a case in which an operationof a device to be operated, particularly a detailed operation such asadjustment of parameters and the like is performed, the burden furtherincreases even if the desired device to be operated is selected.

Meanwhile, utilizing a projected GUI instead of the aforementioned NUImay also be conceivable. For example, projecting a GUI in thesurroundings of a user and operating a device to be operated inaccordance with operations performed to the projected GUI may beexemplified.

However, the difficulty in selecting a device to be operated is stillnot solved even in this case.

Thus, one embodiment of the present disclosure will propose a mechanismcapable of reducing a burden on a user for selecting a device to beoperated that the user desires to operate. Moreover, a mechanism capableof providing operation objects suitable for conditions of the user and amechanism capable of performing an operation regarding a displaylocation of the operation objects in a sense of moving actual objectswill also be proposed.

2. Outline of Information Processing System According to One Embodimentof the Present Disclosure

Next, an outline of an information processing system according to oneembodiment of the present disclosure will be described with reference toFIG. 1. FIG. 1 is a diagram for describing the outline of theinformation processing system according to one embodiment of the presentdisclosure.

The information processing system according to the one embodiment of thepresent disclosure includes an information processing apparatus 100 thathas a user form recognition function, a projection control function, adevice control function, and a communication function, a projectiondevice, an imaging device, and devices to be operated. The user formrecognition function is a function of recognizing a user's body form.The projection control function is a function of controlling a form, aprojection location, and the like of an image that the projection deviceis caused to project. The device control function is a function ofcontrolling processing of the devices to be operated. The communicationfunction is a function of communicating information to apparatuses ordevices outside the information processing apparatus 100. Therefore, theinformation processing apparatus 100 can control devices to be operatedthat are connected via communication in response to an operation using auser's body on the image for operating the devices to be operated thatthe projection device is caused to project (hereinafter, also referredto as operation objects).

Further, the information processing apparatus 100 has adevice-to-be-operated selecting function, an operation object formcontrol function, and an operation object moving function, in additionto the aforementioned functions. The device-to-be-operated selectingfunction is a function of selecting a device to be operated, which is anoperation target, from among a plurality of devices to be operated. Theoperation object display function is a function of controlling a displayform of operation objects to be displayed. The operation object movingfunction is a function of controlling displacement of the operationobjects to be displayed.

As illustrated in FIG. 1, for example, the information processing system1 includes an information processing apparatus 100, a projection imagingdevice 10, a display device 20, and an air conditioner device 21. Theinformation processing apparatus 100 is connected to the projectionimaging device 10, the display device 20, and the air conditioner device21 via a network such as the Internet. Note that the projection imagingdevice 10 may be a projection device and an imaging device as separatedevices.

First, the information processing apparatus 100 selects the displaydevice 20 and the air conditioner device 21 as devices to be operated.Next, the information processing apparatus 100 causes the projectionimaging device 10 to project operation objects for operating theselected display device 20 or air conditioner device 21. In addition,the projection imaging device 10 images a range within which the userfalls and transmits image information related to the image obtained bythe imaging to the information processing apparatus 100. The informationprocessing apparatus 100 recognizes a user's operation from a user'sbody form recognized on the basis of the received image information.Then, the information processing apparatus 100 causes a display locationof the operation objects to move in accordance with the user'soperation. Also, the information processing apparatus 100 controlsprocessing of the display device 20 or the air conditioner device 21 onthe basis of the user's operation performed to the operation objects.

Further, an outline of processing of the information processing system 1will be described with reference to FIG. 2. FIG. 2 is a flowchartillustrating an outline of processing of the information processingsystem 1 according to one embodiment of the present disclosure.

The information processing system 1 selects a device to be operated,which is an operation target (Step S201). Specifically, the informationprocessing apparatus 100 selects a device to be operated, which is anoperation target, from among a plurality of devices to be operated thatare connected via communication.

Next, the information processing system 1 determines whether or not oneor more devices to be operated have been selected (Step S202).Specifically, the information processing apparatus 100 determineswhether or not one or more devices to be operated have been selected asoperation targets.

If it is determined that one or more devices to be operated have beenselected, the information processing system 1 displays operation objects(Step S203). Specifically, the information processing apparatus 100causes the projection imaging device 10 to project operation objects forthe device to be operated, which has been selected as an operationtarget.

Next, the information processing system 1 causes the operation objectsto move (Step S204). Specifically, the information processing apparatus100 causes a projection location of the operation objects to move inaccordance with a user's operation performed to the projected operationobjects.

In addition, the information processing system 1 operates a device to beoperated which is the operation target (Step S205). Specifically, theinformation processing apparatus 100 causes the device to be operatedcorresponding to an operation object to execute processing in accordancewith the user's operation performed to the projected operation object.

Then, the information processing system 1 determines whether or not theoperation has ended (Step S206). Specifically, the informationprocessing apparatus 100 determines whether or not the operation of thedevice to be operated by using the operation object has ended. Note thatif it is determined that the operation has ended, the informationprocessing system 1 causes the display of the operation object to end.

The above is a description regarding the outline of the informationprocessing system according to one embodiment of the present disclosure.Hereinafter, the aforementioned device-to-be-operated selectingfunction, the operation object form control function, and the operationobject moving function will be respectively described in detail indifferent embodiments.

3. First Embodiment of the Present Disclosure (Selection of Device to beOperated)

First, an information processing system 1 according to a firstembodiment of the present disclosure will be described. In the firstembodiment of the present disclosure, an embodiment of theaforementioned device-to-be-operated selecting function in theinformation processing system 1 will mainly be described.

3-1. Configuration of System

A functional configuration of the information processing system 1according to the embodiment will be described with reference to FIG. 3.FIG. 3 is a block diagram schematically illustrating an example of thefunctional configuration of the information processing system 1according to the first embodiment of the present disclosure.

The information processing system 1 includes an information processingapparatus 100-1, a projection imaging device 10, and a display device 20and an air conditioner device 21 as devices to be operated, asillustrated in FIG. 3. Hereinafter, functions of the informationprocessing apparatus 100-1 and the projection imaging device 10 will bedescribed in detail.

(Information Processing Apparatus)

The information processing apparatus 100-1 includes a recognition unit101, a device selection unit 102, a projection control unit 103, adevice control unit 104, a communication unit 105, and a storage unit106.

(Recognition Unit)

The recognition unit 101 recognizes a user's form as an operationsubject. Specifically, the recognition unit 101 recognizes a user's bodyform on the basis of observation information. More specifically, theobservation information is image information related to an image inwhich the user appears, and the recognition unit 101 recognizes theuser's body form by analyzing an image related to the image information.For example, the recognition unit 101 recognizes a user's face or user'seyes that appear in an image and recognizes a user's line of sight onthe basis of the arrangement, the shape, and the like of the recognizedface or eyes. Note that the observation information may be measurementinformation related to a user's motion, position, or the like, and therecognition unit 101 may recognize the user's body form on the basis ofthe measurement information. The body form includes a form of visualrecognition such as a line of sight or a field of view. In addition, themeasurement information may be acquired from a sensor that the userwears, a sensor that is mounted on an object that is present in thesurroundings of the user, or the like.

In addition, the recognition unit 101 recognizes a user's operation onthe basis of the recognized user's form. Specifically, the recognitionunit 101 recognizes an operation performed to a device to be operated onthe basis of the user's form with respect to an operation object thatthe projection control unit 103 is caused to project. For example, if anoperation of touching an operation object is recognized, the recognitionunit 101 recognizes that an operation has been performed to theoperation object.

(Device Selection Unit)

The device selection unit 102 selects a device to be operated which isan operation target (hereinafter, also referred to as an operationtarget device) on the basis of the user's body form). Specifically, thedevice selection unit 102 selects the operation target device on thebasis of the user's body form directed to the device to be operated infirst device selection. For example, the device selection unit 102selects an operation target device on the basis of a user's visualrecognition form directed to the device to be operated. Further, thefirst device selection will be described in detail with reference toFIG. 4. FIG. 4 is a diagram for describing an example of the firstdevice selection through which one device to be operated is selected inthe information processing system 1 according to the embodiment.

The device selection unit 102 decides a device selection range on thebasis of line-of-sight information related to the user's line of sightrecognized by the recognition unit 101. For example, the deviceselection unit 102 decides a range in a real space corresponding to theuser's eyesight as illustrated in FIG. 4 as the device selection range,on the basis of the line-of-sight information provided from therecognition unit 101. Note that the device selection range may besmaller than that of the estimated user's eyesight.

Next, the device selection unit 102 selects a device to be operated thatis determined to fall within the device selection range in accordancewith the user's decision operation. For example, if the user's decisionoperation for the device selection range is recognized by therecognition unit 101, the device selection unit 102 determines whetheror not any device to be operated is present within the decided range.Then, the device selection unit 102 selects the device to be operatedthat is determined to be present within the decided range, for example,the display device 20 as illustrated in FIG. 4 as the operation targetdevice. Note that position information of the device to be operated inthe real space may be provided from the recognition unit 101 or may beprovided from an external device.

In addition, in a case in which a plurality of devices to be operatedare included in the device selection range, the device selection unit102 selects the plurality of devices to be operated as candidates forthe operation target device (hereinafter, also referred to as candidatedevices). Further, the first device selection in a case in which aplurality of devices to be operated are selected as candidate deviceswill be described in detail with reference to FIG. 5. FIG. 5 is adiagram for describing an example of the first device selection throughwhich a plurality of devices to be operated are selected in theinformation processing system 1 according to the embodiment. Note thatdescription of processing that is substantially the same as theaforementioned processing will be omitted.

The device selection unit 102 decides a device selection range on thebasis of line-of-sight information related to the user's line of sightrecognized by the recognition unit 101.

Next, the device selection unit 102 selects devices to be operated thatare determined to fall within the device selection range. For example,the device selection unit 102 determines whether or not any devices tobe operated are present within the decided range. The device selectionunit 102 selects the plurality of devices to be operated that aredetermined to be present within the decided range, for example, all ofthe display device 20, the air conditioner device 21, and a blowerdevice 22 as illustrated in FIG. 5, for example, as candidate devices.

Note that although the example in which all the devices to be operatedthat fall within the device selection range are selected as candidatedevices has been described above, a part of the devices to be operatedthat fall within the device selection range may be selected as candidatedevices. For example, only devices to be operated at distances of equalto or less than a threshold value from the line of sight may be selectedas the candidate devices from among the devices to be operated withinthe device selection range.

In addition, although the example in which the device to be operatedthat is actually present in the region decided from the user's body form(that is, the device selection range) is selected through the firstdevice selection has been described above, a device to be operated thatis linked to the device selection range may be selected. In this case,it is possible to select the device to be operated even in a case inwhich the device to be operated cannot be visually recognized in adirect manner. For example, tags linked to devices to be operated arearranged in a real space, and the device selection unit 102 selects adevice or devices to be operated linked to a tag or tags within thedevice selection range as candidate devices or an operation targetdevice. In addition, for example, a specific region in the real space islinked to a device or devices to be operated, and the device selectionunit 102 selects the device or devices to be operated linked to thespecific region within the device selection range as candidate devicesor an operation target device. When the aforementioned tags areprovided, the user can reliably recognize the device to be operated andthen perform the first device selection. Also, when the aforementionedspecific region is provided, it is possible to omit the aforementionedtags and to reduce efforts or costs for prior preparation or changing inthe linking.

Further, the device selection unit 102 selects an operation targetdevice from among candidate devices selected through the first deviceselection. Specifically, the device selection unit 102 selects anoperation target device from among the candidate devices selectedthrough the first device selection on the basis of the user's selectingoperation performed to selection objects projected by the projectionimaging device 10, in second device selection. Further, the seconddevice selection will be described in detail with reference to FIG. 6.FIG. 6 is a diagram for describing an example of the second deviceselection through which an operation target device is selected on thebasis of an operation performed to a selection object in the informationprocessing system according to the embodiment.

First, information indicating a plurality of candidate devices selectedthrough the first device selection is presented as selection objects tothe user after the first device selection. For example, selectionobjects 31A, 32A, and 33A respectively corresponding to the blowerdevice 22, the display device 20, and the air conditioner device 21 asillustrated in FIG. 6, which have been selected through the first deviceselection, are respectively projected in a selection object displayregion 30 by the projection imaging device 10.

Next, if an operation of selecting one candidate device from among theplurality of candidate devices is recognized by the recognition unit101, the device selection unit 102 selects the selected candidate deviceas an operation target device. For example, if an operation of touchingthe selection object 32A is recognized by the recognition unit 101, thedevice selection unit 102 selects the display device 20 corresponding tothe selected selection object 32A as the operation target device.

(Projection Control Unit)

The projection control unit 103 serves as a display control unit andcontrols projection performed by the projection imaging device 10.Specifically, the projection control unit 103 controls projection ofselection objects related to candidate devices selected through thefirst device selection. More specifically, the projection control unit103 causes the projection imaging device 10 to project selection objectsindicating the candidate devices selected through the first deviceselection. Further, selection objects will be described in detail withreference to FIGS. 6 and 7. FIG. 7 is a diagram for describing anexample of priority information in the information processing system 1according to the embodiment.

If a plurality of candidate devices are selected through the firstdevice selection, the projection control unit 103 decides forms ofselection objects for the plurality of candidate devices on the basis ofpriority information. The priority information includes informationdecided on the basis of a body form in the first device selection. Morespecifically, the projection control unit 103 decides arrangement of theselection objects on the basis of a user's line of sight in the firstdevice selection. For example, the projection control unit 103 decidesforms of the selection objects in accordance with distances from theuser's line of sight to the candidate devices in a three-dimensionalspace in the first device selection. In detail, the projection controlunit 103 calculates a distance d1 from the user's line of sight to theblower device 22, a distance d2 from the user's line of sight to the airconditioner device 21, a distance d3 from the user's line of sight tothe display device 20 as illustrated in FIG. 7. Then, the projectioncontrol unit 103 decides alignment of the selection object in anascending order or a descending order of the calculated distances. Notethat it is a matter of course that the distances from the line of sightto the candidate devices may be distances in a two-dimensional space. Inaddition, the selection objects may be arranged at locations that becomecloser to the user as priority becomes higher, that is, as theaforementioned distances become shorter.

In addition, the projection control unit 103 decides a location inaccordance with the user's decision operation in the first deviceselection as a projection location of the selection objects. Thelocation in accordance with the decision operation includes a user'sbody part designated by the decision operation. In a case in which a tapoperation as a decision operation for a device selection range in thefirst device selection is performed to the user's thigh, for example,the projection control unit 103 decides a region 30 on the user's thighas a projection location of the selection objects. Note that theprojection control unit 103 may decide the display location of theselection objects in accordance with an operation of designating thedisplay location of the selection objects, which is different from theaforementioned decision operation in the first device selection.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the selection objects in a decided form at thedecided projection location. For example, the projection control unit103 causes the projection imaging device 10 to project the selectionobjects 31A, 32A, and 33A in a form of a list of decided alignment inthe region 30 on the user's thigh as illustrated in FIG. 6 decided asthe projection location.

Note that selection objects may be projected for devices to be operatedrelated to the candidate devices selected through the first deviceselection. Specifically the projection control unit 103 grasps thedevices to be operated that operate in conjunction with the candidatedevices and causes the projection imaging device 10 to project theselection objects for the grasped devices to be operated along withselection objects for the candidate devices. For example, selectionobjects for a recording device, a sound output device, an illuminationdevice, or the like that operates in conjunction with the display device20 may be projected. In addition, selection objects for devices to beoperated that have functions that are similar to functions of thecandidate devices may be projected. In this case, it is possible toimprove usability by selection objects for devices that the user islikely to desire to operate being projected.

Further, the projection control unit 103 may serve as a notificationcontrol unit and control a notification of the devices to be operated,which have been selected through the first device selection, when theselection objects are projected. Specifically, the projection controlunit 103 controls projection indicating linkage between the devices tobe operated, which have been selected through the first deviceselection, and the selection objects. Further, the notification of thedevices to be operated, which have been selected through the firstdevice selection, will be described in detail with reference to FIG. 8.FIG. 8 is a diagram illustrating an example of the notification of thedevices to be operated, which have been selected through the firstdevice selection, in the information processing system 1 according tothe embodiment.

If the decision operation in the first device selection is recognized,the projection control unit 103 causes the projection imaging device 10to project display indicating linkage between a plurality of respectivecandidate devices selected through the first device selection and thelocation at which the decision operation has been performed.Specifically, if the decision operation is recognized as having beenperformed to the user's thigh in the first device selection, animationof an image (including simple light) with a track that follows a lineconnecting each of the display device 20, the air conditioner device 21,and the blower device 22 with the region 30 on the user's thigh inaccordance with the decision operation is projected. For example,animation in which an image moves from the candidate devices to theregion 30 may be projected, and the selection objects may be projectedwhen the image reaches the region 30.

The projection control in the selection of devices has been describedabove. In addition, the projection control unit 103 controls projectionof operation objects for a device to be operated, which has beenselected through the second device selection. Specifically, if aselecting operation of a candidate device is recognized in the seconddevice selection, the projection control unit 103 causes the projectionimaging device 10 to project operation objects for the selectedcandidate device. For example, if a selecting operation performed to aselection object is recognized, the projection of the selection objectsis caused to end, and the operation objects may be projected at thelocation at which the selection objects have been projected.

(Device Control Unit)

The device control unit 104 controls a device to be operated.Specifically, the device control unit 104 controls processing of thedevice to be operated on the basis of a user's operation recognized bythe recognition unit 101. For example, the device control unit 104decides processing of the display device 20 in accordance with anoperation of an operation object for the display device 20 and causesthe communication unit 105 to transmit a processing execution requestfor requesting execution of the decided processing to the display device20.

(Communication Unit)

The communication unit 105 communicates with devices outside theinformation processing apparatus 100-1. Specifically, the communicationunit 105 transmits image information to the projection imaging device 10and receives image information from the projection imaging device 10. Inaddition, the communication unit 105 transmits a processing executionrequest to the display device 20 and the air conditioner device 21. Notethat the communication unit 105 may perform communication in either awired communication scheme or a wireless communication scheme.

(Storage Unit)

The storage unit 106 stores information that is used in processing ofthe information processing apparatus. Specifically, the storage unit 106stores information that is used for analyzing observation information inrecognition processing performed by the recognition unit 101. Inaddition, the storage unit 106 stores image information related to animage that the projection control unit 103 causes the projection imagingdevice 10 to project. Note that information stored in an external devicemay be acquired via communication instead of the information beingstored in the storage unit 106.

(Projection Imaging Device)

The projection imaging device 10 projects an image on the basis of aninstruction from the information processing apparatus 100-1.Specifically, the projection imaging device 10 projects an image relatedto image information that is provided from the information processingapparatus 100-1 to a designated location. For example, the projectionimaging device 10 may be a projector capable of rotating a projectiondirection with respect to two axes. Note that the projection imagingdevice 10 may be an omnidirectional projector, a hologram video device,or a display device that an object arranged in the surroundings of theuser (for example, a table or a sofa) has. In addition, the projectionimaging device 10 may project different images to a plurality oflocations at the same time.

In addition, the projection imaging device 10 images the surroundings ofthe device itself. Specifically, the projection imaging device 10 imagesthe surroundings of the device itself at predetermined time intervals orin response to a request from the information processing apparatus100-1. Then, the projection imaging device 10 transmits imageinformation related to the image acquired by the imaging to theinformation processing apparatus 100-1. Note that an imaging availablerange may be the same as a projection available range or may be broaderthan the projection available range. In addition, an imaging range maybe caused to conform to a projection range. Also, there may be aplurality of imaging ranges.

3-2. Processing of System

Next processing of the information processing system 1 according to theembodiment will be described.

(Overall Processing)

First, overall processing of the information processing system 1 will bedescribed with reference to FIG. 9. FIG. 9 is a flowchart conceptuallyillustrating an example of the overall processing of the informationprocessing system 1 according to the embodiment.

The information processing system 1 estimates a user's body form (StepS301). Specifically, the recognition unit 101 recognizes the user's bodyform by using image information or the like.

Next, the information processing system 1 determines whether or not adecision operation in the first device selection has been performed(Step S302). Specifically, the recognition unit 101 decides a deviceselection range on the basis of the recognized user's body form. Inaddition, the recognition unit 101 attempts to recognize the decisionoperation in the first device selection on the basis of the recognizeduser's body form. Then, the device selection unit 102 determines whetheror not the decision operation in the first device selection has beenrecognized by the recognition unit 101.

If it is determined that the decision operation in the first deviceselection has been performed, the information processing system 1determines whether or not devices to be operated have been selected(Step S303). Specifically, if it is determined that the decisionoperation in the first device selection has been recognized by therecognition unit 101, the device selection unit 102 determines whetheror not one or more devices to be operated are present in the deviceselection range.

If it is determined that the devices to be operated have been selected,the information processing system 1 determines whether or not aplurality of devices to be operated have been selected (Step S304).Specifically, the device selection unit 102 determines whether or nottwo or more devices to be operated are present within the deviceselection range.

If it is determined that a plurality of devices to be operated have beenselected, the information processing system 1 displays selection objects(Step S305). Specifically, if it is determined that two or more devicesto be operated are present, the device selection unit 102 selects thetwo or more devices to be operated as candidate devise. Then, theprojection control unit 103 causes the communication unit 105 totransmit image information related to selection objects for thecandidate devices to the projection imaging device 10. Then, theprojection imaging device 10 projects the selection objects related tothe received image information to a designated location. Note thatdetails thereof will be described later.

Next, the information processing system 1 determines whether or not adecision operation in the second device selection has been performed(Step S306). Specifically, the recognition unit 101 attempts torecognize the decision operation in the second device selection. Then,the device selection unit 102 determines whether or not the decisionoperation in the second device selection has been recognized by therecognition unit 101.

If it is determined that the decision operation in the second deviceselection has been performed, the information processing system 1displays operation objects for the device to be operated (Step S307).Specifically, if it is determined that the decision operation in thesecond device selection has been recognized by the recognition unit 101,the device selection unit 102 causes the communication unit 105 totransmit image information related to the operation objects to theprojection imaging device 10. Then, the projection imaging device 10projects the operation objects related to the received image informationinstead of the selection objects.

(Overall Processing)

Subsequently, selection object display processing of the informationprocessing system 1 will be described with reference to FIG. 10. FIG. 10is a flowchart conceptually illustrating an example of the selectionobject display processing of the information processing system 1according to the embodiment.

The information processing system 1 acquires user's body forminformation (Step S311). Specifically, the projection control unit 103acquires information related to a user's body form recognized by therecognition unit 101.

Next, the information processing system 1 decides a form of selectionobjects on the basis of the body form information (Step S312).Specifically, the projection control unit 103 decides priorityinformation on the basis of the acquired body form information. Then,the projection control unit 103 decides alignment of the selectionobjects on the basis of the priority information.

In addition, the information processing system 1 decides a displaylocation in accordance with a decision operation in the first deviceselection (Step S313). Specifically, the projection control unit 103decides the location at which the decision operation in the first deviceselection has been performed as the display location of the selectionobjects.

Then, the information processing system 1 displays the selection objectsin the decided form at the decided location (Step S314). Specifically,the projection control unit 103 causes the communication unit 105 totransmit image information related to the selection objects in thedecided alignment along with an instruction of the projection locationto the projection imaging device 10. Then, the projection imaging device10 projects the selection objects related to the received imageinformation to the designated location.

3-3. Summary of First Embodiment

In this manner, according to the first embodiment of the presentdisclosure, the information processing system 1, that is, theinformation processing apparatus 100-1 controls display of selectionobjects related to devices to be operated, which have been selectedthrough the first device selection based on information from which abody form of an operation subject directed to the devices to be operatedis estimated. Then, the information processing apparatus 100-1 controlsa device to be operated, which has been selected through the seconddevice selection based on information related to a selecting operationperformed by the operation subject to the selection objects.

Conventionally, either a first interface that operates the device to beoperated in proximity to the device or a second interface that operatesthe device to be operated in a remote manner has mainly be provided.However, the user has to move up to the device to be operated accordingto the first interface. Also, it is difficult to accurately select thedevice to be operated with an NUI operation such as a gesture operationin the second interface. In addition, in a case in which a plurality ofdevices to be operated are present, efforts to find remote controllerscorresponding to the devices to be operated occur in a remote controlleroperation in the second interface.

Meanwhile, according to the embodiment, candidates for a device to beoperated, which is selected through the first device selection based ona user's body form, are presented to the user, and the user can select adevice to be operated from among the candidates. Therefore, it is notnecessary for the user to move to the device to be operated first. Inaddition, it is possible to suppress improper selection of the device tobe operated and to prevent reselection of the device to be operated bythe user selecting an operation target from among the presentedcandidates for the device to be operated. Also, it is possible tooperate the device to be operated without any specific device such as aremote controller and to suppress occurrence of efforts to find theremote controller or the like by the device to be operated beingoperated on the basis of selection from among selection objects.Therefore, it is possible to reduce a burden on the user for selectingthe device to be operated that the user desires to operate.

In addition, the aforementioned selection objects include objectsindicating the devices to be operated, which have been selected throughthe aforementioned first device selection. Therefore, it is possible forthe user to reliably select the device to be operated that the userhimself/herself intends by the candidate devices selected through thefirst device selection being explicitly indicated.

In addition, the aforementioned selection objects are displayed suchthat the selection objects are visually recognized on a form based onthe priority information. Although a plurality of candidate devices areselected through the first device selection, the user generally desiresto operate one device to be operated in practice. Thus, it is possibleto improve operability of the selection objects by causing theprojection imaging device 10 to project the selection objects such thatthe desired device to be operated is easily selected. In addition, aform of the selection objects controlled on the basis of priorityinformation includes arrangement of the selection objects. Therefore, itis possible to prompt the user to intuitively grasp the device to beoperated that the user desires to operate. Accordingly, it is possibleto improve the operability of the selection objects.

In addition, the aforementioned priority information includesinformation decided on the basis of information from which theaforementioned user's body form in the aforementioned first deviceselection is estimated. The user has already decided the device to beoperated that the user desires to operate at the timing of the firstdevice selection. Therefore, the selection objects for the desireddevice to be operated are more likely to be selected, and theoperability can be further improved by the form of the selection objectsbeing decided in accordance with how high the probability of being thedesired device to be operated, which estimated from the body form in thefirst device selection, is.

In addition, the information processing apparatus 100-1 controls displayof operation objects for the device to be operated, which has beenselected through the aforementioned second device selection. Therefore,it is possible to operate the desired device to be operated inaccordance with user's intention. Therefore, it is possible to improveusability in the operation of the selected device to be operated.

In addition, the aforementioned selection objects are displayed at alocation in accordance with a decision operation performed by theaforementioned operation subject in the first device selection, inaccordance with the decision operation. Therefore, it is possible topromote selection of the desired device to be operated as a candidatedevice by the first device selection being performed in accordance withthe user's intention. In addition, it is possible to improve theoperability of the selection objects by the selection objects beingprojected at a location that the user intends. In addition, the locationin accordance with the aforementioned decision operation includes a bodypart of the operation subject that is designated through theaforementioned decision operation. Therefore, it is possible to projectthe selection objects at a location at which the user can easily operatethe selection objects even in a case in which a projection space for theselection objects is not secured in the surroundings of the user, by theselection objects being projected to the user's body. Also, in a case inwhich the projection location is tracked, it is possible to continuouslyoperate the projected selection objects since the selection objects moveeven if the user moves.

In addition, the information processing apparatus 100-1 controls anotification of the devices to be operated, which have been selectedthrough the aforementioned first device selection, when theaforementioned selection objects are displayed. Therefore, it ispossible for the user to check the candidate devices selected throughthe first device selection. In a case in which the desired device to beoperated has not been selected, for example, the user can retry thefirst device selection. Therefore, it is possible to suppress improperselection of the operation target device and to improve efficiency inthe selection of the operation target device.

In addition, the aforementioned notification includes a display outputindicating linkage between the aforementioned selected device to beoperated and the aforementioned selection objects. Therefore, it ispossible to prompt the user to check the selected candidate devices andto guide the user to the selection objects. In a case in which the firstdevice selection is performed on the basis of a user's visualrecognition form, in particular, it is possible to guide the user's lineof sight from the devices to be operated to the selection objects.Therefore, it is possible to smoothly guide the user from the firstdevice selection to the second device selection and to make it easy toselect the operation target device.

In addition, the aforementioned body form includes a visual recognitionform of the aforementioned operation subject, and devices to be operatedthat are determined to fall within at least a part of the eyesight ofthe operation subject are selected through the aforementioned firstdevice selection. Therefore, it is possible to select the devices to beoperated without moving a user's body part such as a hand or a leg ofthe user. Therefore, the user can select a device to be operated thatthe user desires to operate and operate the device while performinganother task or the like. In addition, in a case in which the useroperates the device to be operated, the user's line of sight isgenerally directed to the device to be operated. Therefore, it ispossible to improve the probability that the candidate device is thedevice to be operated that the user desires to operate, by the firstdevice selection being performed on the basis of the visual recognitionform.

3-4. Modification Examples

The first embodiment of the present disclosure has been described above.Note that the embodiment is not limited to the aforementioned example.Hereinafter, first to fifth modification examples of the embodiment willbe described.

First Modification Example

In a first modification example of the embodiment, the informationprocessing system 1 may perform the first device selection on the basisof another body form. Specifically, the information processing apparatus100-1 performs the first device selection on the basis of a user'sposture. More specifically, the device selection unit 102 selects adevice or devices to be operated that is determined to fall within arange decided from the user's posture in the first device selection asan operation target device or candidate devices. Further, processing inthe modification example will be described with reference to FIG. 11.FIG. 11 is a diagram for describing an example of the first deviceselection in the information processing system 1 according to the firstmodification example of the embodiment.

The recognition unit 101 recognizes a posture of at least a part of theuser's body on the basis of image information or the like. For example,the recognition unit 101 recognizes an orientation of the user's face orbody that appears in an image on the basis of image information relatedto a three-dimensional image received from the projection imaging device10.

Next, the device selection unit 102 decides a device selection range onthe basis of the user's posture. For example, the device selection unit102 decides a device selection range as illustrated in FIG. 11 withreference to the recognized orientation of the user's face or body.

Then, the device selection unit 102 selects a device or devices to beoperated that falls within the decided device selection range as anoperation target device or candidate devices. For example, the deviceselection unit 102 selects the display device 20 that falls within thedecided device selection range as illustrated in FIG. 11 as an operationtarget device. Note that in a case in which a plurality of devices to beoperated fall within the decided device selection range, the pluralityof devices to be operated are selected as candidate devices.

In addition, the information processing apparatus 100-1 may perform thefirst device selection on the basis of user's motion in another example.Specifically, the device selection unit 102 selects a device or devicesto be operated that are determined to fall within a range decided fromuser's motion in the first device selection as an operation targetdevice or candidate devices. Further, processing in this another exampleof the modification example will be described with reference to FIG. 12.FIG. 12 is a diagram for describing another example of the first deviceselection in the information processing system 1 according to the firstmodification example of the embodiment.

The recognition unit 101 recognizes user's motion on the basis of imageinformation or the like. For example, the recognition unit 101recognizes a user's gesture or action that appears in an image on thebasis of image information related to a three-dimensional image receivedfrom the projection imaging device 10. The gesture includes, forexample, a motion of depicting a circle, double tapping, flicking,clapping, contact to an object, and the like.

Next, the device selection unit 102 decides a device selection range onthe basis of user's motion. If gripping of a hand directed to a deviceto be operated as illustrated in FIG. 12 is recognized, for example, thedevice selection unit 102 decides the device selection range asillustrated in FIG. 12 with reference to the hand.

Then, the device selection unit 102 selects a device or devices to beoperated that falls within the decided device selection range as anoperation target device or candidate devices. Specifically, the deviceselection unit 102 selects the display device 20 that falls within thedecided device selection range as illustrated in FIG. 12 as an operationtarget device. Note that in a case in which a plurality of devices to beoperated fall within the decided device selection range, the pluralityof devices to be operated are selected as candidate devices.

In addition, the information processing apparatus 100-1 may perform thefirst device selection on the basis of user's speech generation in yetanother example. Specifically, the device selection unit 102 selects adevice or devices to be operated that are determined to fall within arange decided from user's speech generation in the first deviceselection as an operation target device or candidate devices. Further,processing in this yet another example of the modification example willbe described with reference to FIG. 13. FIG. 13 is a diagram fordescribing a yet another example of the first device selection in theinformation processing system 1 according to the first modificationexample of the embodiment.

The recognition unit 101 recognizes user's speech generation on thebasis of sound information. For example, the recognition unit 101recognizes presence of user's speech generation or content of speechgeneration on the basis of sound information received from a sound inputdevice that the information processing system 1 separately includes.

Next, the device selection unit 102 decides a device selection range onthe basis of user's speech generation. If content of user's speechgeneration of “living room” as illustrated in FIG. 13 is recognized, forexample, the device selection unit 102 decides the living room as adevice selection range.

Then, the device selection unit 102 selects a device or devices to beoperated that fall within the decided device selection range as anoperation target device or candidate devices. Specifically, the deviceselection unit 102 selects the display device 20, the air conditionerdevice 21, and the blower device 22 that are present in the decideddevice selection range, namely in the living room as candidate devices.Note that in a case in which only one device to be operated is presentin the decided device selection range, the device to be operated isselected as an operation target device.

According to the first modification example of the embodiment, the bodyform of the operation subject includes a posture of the operationsubject, and the devices to be operated that are determined to fallwithin a region decided from the posture of the operation subject areselected through the first device selection as described above. In acase in which the user operates a device to be operated, the user isgenerally in a posture in accordance with the device to be operated.Therefore, it is possible to select appropriate candidate devices in thefirst device selection by selecting the candidate devices in accordancewith the user's posture. In addition, since there is a case in whichprocessing with a higher burden is used in the line-of-sight detectionthan in the posture detection, it is possible to reduce the processingburden as compared with the first device selection based on the visualrecognition form. Therefore, it is possible to improve responsiveness.

In addition, the body form of the operation subject includes motion ofthe operation subject, and devices to be operated that are determined tofall within a region decided from the motion of the operation subjectare selected through the first device selection in another example ofthe modification example. Therefore, it is possible to realize the firstdevice selection that further conforms to user's intention by selectionfrom among the candidate devices in accordance with user's explicitmotion with respect to the first device selection. Therefore, it ispossible to further improve operability in the first device selection.

In addition, the body form of the operation subject includes speechgeneration from the operation subject, and devices to be operated thatare determined to fall within a region decided from the speechgeneration from the operation subject are selected through the firstdevice selection in yet another example of the modification example.Therefore, it is possible for the user to select a device to be operatedwithout moving his/her body. Therefore, it is possible to select adevice to be operated that the user desires to operate and to operatethe device while separately performing a task and without directing theline of sight to the device.

Note that the first device selection may be performed on the basis of aform of an object that the user operates, instead of the user's body.Specifically, the information processing apparatus 100-1 decides adevice selection range from a form of an object that the user operatesand selects devices to be operated, which fall within the deviceselection range in accordance with a decision operation performed to theobject in the first device selection.

For example, the object that the user operates may be a device such as asmartphone, and the recognition unit 101 recognizes an orientation orthe like of the smartphone. The device selection unit 102 decides adevice selection range in accordance with the recognized orientation orthe like of the smartphone. Then, the recognition unit 101 recognizes anoperation, such as a down-flicking operation, performed to the smartphone. If the down-flicking operation is recognized, the deviceselection unit 102 selects devices to be operated in the decided deviceselection range. Note that the recognition unit 101 may acquireinformation related to the form of the object from the object or anotherexternal device instead of recognizing the form of the aforementionedobject from image information or the like.

In this case, it becomes easier to determine whether or not the user'sform is a form for the first device selection as compared with theuser's body form. Therefore, it is possible to suppress the concern ofimproper operating of the processing in the first device selection. Inaddition, in a case in which information related to the form of theobject is obtained from the aforementioned object, it is possible tomore accurately recognize the form of the object. Therefore, it ispossible to improve accuracy of the processing in the first deviceselection.

Second Modification Example

In a second modification example of the embodiment, selection objectsmay be displayed without a decision operation in the first deviceselection. Specifically, the projection control unit 103 decides thebody of the operation subject or the surroundings of the operationsubject that has a region in which the selection objects can bedisplayed such that the selection objects are visually recognized by theoperation subject (hereinafter, also referred to as a projectionavailable region), and causes the projection imaging device 10 toproject the selection objects at the decided location.

More specifically, the projection control unit 103 searches for theprojection available region such that the selection objects are visuallyrecognized by the user. The projection available region may be a regionwith the maximum broadness on the user's body or in the surroundings ofthe user. In addition, the projection available region may be determinedin accordance with a degree of irregularity of the surface, a surfacecolor, a texture, presence of a pattern, or the like. For example, theprojection available region may be a flat surface or a surface withirregularity within an allowable range, a surface with color or textureuniformity within an allowable range, or a surface that does not includeany pattern. In detail, a white flat surface may be selected as theprojection available region with priority.

Further, processing in the modification example will be described withreference to FIG. 14. FIG. 14 is a flowchart conceptually illustratingan example of selection object display processing of the informationprocessing system 1 according to the second modification example of theembodiment. Note that description of processing that is substantiallythe same as the aforementioned processing will be omitted.

The information processing system 1 acquires body form information (StepS321) and decides a form of the selection object on the basis of theacquired body form information (Step S322).

Next, the information processing system 1 decides a display location inaccordance with a display available region (Step S323). Specifically,the projection control unit 103 searches for a projection availableregion on the user's body or in the surroundings of the user. Then, if aregion that satisfies conditions is found, the projection control unit103 decides the region as the projection available region.

Then, the information processing system 1 displays selection objects ina decided form at the decided location (Step S324). Specifically, theprojection control unit 103 causes the projection imaging device 10 toproject the selection objects in the region on the user's body or in thesurroundings of the user decided as the projection available region.

According to the second modification example of the embodiment, theselection objects are displayed on the body of the operation subject orin the surroundings of the operation subject that have a region in whichthe selection objects can be displayed such that the selection objectsare visually recognized by the operation subject as described above.Therefore, it is possible to project the selection objects without theuser's decision operation in the first device selection. Therefore, itis possible to perform the first device selection even in a situation inwhich the user cannot perform the decision operation, for example, whenthe user is performing a task.

Third Modification Example

In a third modification example of the embodiment, the display form ofthe selection objects may be another form. Specifically, the displaylocation of the selection objects may be another location. For example,the selection objects may be displayed on a display unit designatedthrough a decision operation in the first device selection. In detail,in a case in which an operation destination of the decision operation inthe first device selection is the display device, the projection controlunit 103 causes the display device to display the selection objects.Further, processing in the modification example will be described withreference to FIG. 15. FIG. 15 is a diagram illustrating an example ofdisplay of the selection objects in the information processing system 1according to the third modification example of the embodiment.

The recognition unit 101 recognizes an operation destination of thedecision operation in the first device selection. If the decisionoperation in the first device selection is recognized, for example, therecognition unit 101 recognizes the operation destination of thedecision operation.

In a case in which the recognized operation destination is the displaydevice, the projection control unit 103 determines whether or not it ispossible to control the display device. In a case in which therecognized operation destination is a smartphone 70 as illustrated inFIG. 15, for example, the projection control unit 103 determines whetheror not it is possible to control at least a display function of thesmartphone 70.

In a case in which it is determined to be possible to control thedisplay device, the projection control unit 103 causes the displaydevice to display the selection objects. If it is determined to bepossible to control the display function of the smartphone 70, forexample, the projection control unit 103 causes the communication unit105 to transmit image information related to the selection objects tothe smartphone 70. The smartphone 70 displays selection objects 31A to33A on the display unit on the basis of the received image information.

Note that if a selecting operation on the selection objects displayed onthe display unit of the smartphone 70 is performed, operation objectsfor the device to be operated corresponding to the selected selectionobject is displayed on the display unit of the smartphone 70. Then, theoperation target device is controlled in accordance with an operationperformed to the displayed operation objects.

In addition, the form of the selection objects in accordance withpriority information may be another form. Specifically, the sizes of theselection objects may be decided in accordance with the priorityinformation. For example, the projection control unit 103 decides thesizes of the selection objects on the basis of the body form in thefirst device selection. Further, other processing in the modificationexample will be described with reference to FIG. 16. FIG. 16 is adiagram illustrating another example of display of the selection objectsin the information processing system 1 according to the thirdmodification example of the embodiment.

The projection control unit 103 decides the sizes of the selectionobjects in accordance with distances from the user's line of sight tothe candidate devices in the three-dimensional space in the first deviceselection. For example, the projection control unit 103 respectivelycalculates the distances from the distances from the user's line ofsight to the display device 20, the air conditioner device 21, and theblower device 22. Then, the projection control unit 103 decides thesizes of the selection objects such that the sizes increase as thecalculated distances are shorter. In detail, the projection control unit103 decides the size of the selection object 32B for the display device20 with the shortest distance calculated from among the candidatedevices as the largest size from among the candidate devices asillustrated in FIG. 16. In addition, the projection control unit 103decides the size of the selection object 31B for the blower device 22with the longest distance calculated from among the candidate devices asthe smallest size from among the candidate devices as illustrated inFIG. 16.

According to the third modification example of the embodiment, thelocation in accordance with the decision operation in the first deviceselection includes the display unit designated through the decisionoperation as described above. Therefore, it is possible to securevisibility of the selection objects by the selection objects beingdisplayed on the display unit. In a case in which the selection objectsare projected, in particular, the configuration according to themodification example is thus advantageous since it becomes difficult toproject the selection objects when some object is present between theprojection location and the projection device.

In addition, the form of the selection objects controlled on the basisof the priority information includes the sizes of the selection objects.Therefore, it is possible to make it easier to grasp the device to beoperated that the user desires to operate. Accordingly it is possible toimprove operability of the selection objects.

Fourth Modification Example

In a fourth modification example of the embodiment, the priorityinformation related to the display form of the selection objects may beother information. Specifically, the priority information may beinformation decided on the basis of biological information of theoperation subject. More specifically, the biological informationincludes information related to user's pulse, body temperature,sweating, brain waves, and the like, and the projection control unit 103estimates a device to be operated that the user desires to operate fromthe biological information. Then, the projection control unit 103decides alignment or sizes of the selection objects such that theselection objects for the estimated device to be operated are easilyselected. In a case in which the user's body temperature is lower than anormal temperature, that is, in a case in which it is estimated that theuser is feeling cold, for example, selection objects are projected withsuch alignment or sizes that selection objects for a candidate devicesuch as an air conditioner device or a heating device are easilyselected.

In addition, the priority information may be information decided on thebasis of information related to a surrounding environment of theoperation subject (hereinafter, also referred to as surroundingenvironment information). For example, the surrounding environmentinformation includes information related to a temperature, a humidityluminance, noise, or the like, and the projection control unit 103estimates a device to be operated that the user desires to operate fromthe surrounding environment information. In a case in which the volumeof noise is higher than a threshold value, that is, in a case in whichit is estimated that the user is feeling noisy, for example, selectionobjects are projected with such alignment or sizes that selectionobjects for a candidate device such as a video replay device or a soundoutput device are easily selected.

Further, processing in the modification example will be described withreference to FIG. 17. FIG. 17 is a flowchart conceptually illustratingan example of selection object display processing of the informationprocessing system 1 according to the fourth modification example of theembodiment. Note that description of processing that is substantiallythe same as the aforementioned processing will be omitted.

The information processing system 1 determines whether or not biologicalinformation or surrounding environment information has been acquired(Step S331). Specifically, the projection control unit 103 determineswhether or not biological information or surrounding environmentinformation has been acquired. Note that the biological information maybe acquired via the communication unit 105 while the surroundingenvironment information may be acquired via the communication unit 105or may be generated in recognition processing of the recognition unit101.

If it is determined that the biological information or the surroundingenvironment information has bee acquired, the information processingsystem 1 decides a form of selection objects on the basis of thebiological information or the surrounding environment information (StepS332). Specifically, the projection control unit 103 estimates a deviceto be operated that the user desires to operate from among candidatedevices from the biological information or the surrounding environmentinformation for the respective candidate devices. Then, the projectioncontrol unit 103 decides the form of the selection object for thecandidate device that is estimated to be the device to be operated thatthe user desires to operate as easily selectable form.

In addition, if it is determined that neither the biological informationnor the surrounding environment information has been acquired, theinformation processing system 1 acquires body form information (StepS333) and decides a form of the selection objects on the basis of theacquired body form information (Step S334).

Next, the information processing system 1 decides a display location inaccordance with the decision operation in the first device selection(Step S335) and displays the selection objects in the decided form atthe decided location (Step S336).

In addition, the priority information may be information decided on thebasis of information related to operations of devices to be operated inthe past. Specifically, the projection control unit 103 acquires, fromthe storage unit 106, a user's operation history of devices to beoperated in the past, which has been recognized by the recognition unit101, and estimates a device to be operated that the user desires tooperate from the operation history. For example, a probability at whichthe user desires to operate a candidate device is estimated on the basisof time zones and locations of operations and operation orders ofdevices to be operated, which are grasped from the operation history, atime zone, a use's location, and a candidate device that is beingselected at a current timing. Then, the selection objects are projectedwith such alignment or sizes that the selection objects with higherprobability estimated are more easily selected.

Further, processing in the modification example will be described withreference to FIG. 18. FIG. 18 is a flowchart conceptually illustratinganother example of selection object display processing of theinformation processing system 1 according to the fourth modificationexample of the embodiment. Note that description of processing that issubstantially the same as the aforementioned processing will be omitted.

The information processing system 1 determines whether or not operationhistories of devices to be operated are present (Step S341).Specifically, the projection control unit 103 determines whether or notoperation histories of the devices to be operated are present in thestorage unit 106. Note that presence of operation histories of thecandidate devices from among operation histories of the devices to beoperated may be determined.

If it is determined that operation histories are present, theinformation processing system 1 decides a form of selection objects onthe basis of the operation histories (Step S342). Specifically, theprojection control unit 103 estimates probabilities at which thecandidate devices are the device to be operated that the user desires tooperate on the basis of the operation histories for the respectivecandidate devices. Then, the projection control unit 103 decides theform of the selection objects for the candidate devices as such a formthat the devices to be operated with higher probabilities at which thecandidate devices are the device to be operated that the user desires tooperate are more easily selected.

In addition, if it is determined that neither the biological informationnor the surrounding environment information has been acquired, theinformation processing system 1 acquires body form information (StepS343) and decides a form of the selection objects on the basis of theacquired body form information (Step S344).

Next, the information processing system 1 decides a display location inaccordance with the decision operation in the first device selection(Step S345) and displays the selection objects in the decided form atthe decided location (Step S346).

According to the fourth modification example of the embodiment, thepriority information includes information decided on the basis of thebiological information of the operation subject or the informationrelated to the surrounding environment of the operation subject asdescribed above. Therefore, it is possible to decide the form of theselection objects in accordance with user's body conditions or bodyfeeling. Accordingly, the selection objects for the desired device to beoperated are more easily selected, and it is possible to further improveoperability.

In addition, the priority information includes information decided onthe basis of information related to operations of the devices to beoperated in the past. Therefore, it is possible to decide a form of theselection objects in accordance with a tendency, a habit, or the like ofuser's operations. Therefore, the probability that the selection objectsthat are displayed in an easily selectable manner are the selectionobjects for the desired device to be operated increases, and it ispossible to further improve operability.

Note that a plurality of sets of priority information pieces may becombined. For example, the projection control unit 103 may decide theform of the selection objects by using a combination of at least two ofthe user's body form information, the biological information, thesurrounding environment information, and the operation histories. Inaddition, the priority information may be transformed into scores or thelike, and the selection objects may be displayed in such a form that thecandidate devices with higher scores are more easily selected.

Fifth Modification Example

In a fifth modification example of the embodiment, the notification ofthe candidate devices may be realized by another method. Specifically,the projection control unit 103 causes the projection imaging device 10to perform such projection that the devices to be operated, which havebeen selected through the first device selection, that is, the candidatedevices or the surroundings of the candidate devices appeardistinctively when the selection objects are displayed. Further, anexample of the notification of the candidate devices in the modificationexample will be described with reference to FIG. 19. FIG. 19 is adiagram illustrating an example of the notification of the devices to beoperated, which have been selected through the first device selection,in the information processing system 1 according to the fifthmodification example of the embodiment.

If a decision operation in the first device selection is recognized, theprojection control unit 103 causes the projection imaging device 10 toperform projection that is visually recognized such that the pluralityof respective candidate devices selected through the first deviceselection or the surroundings of the plurality of respective candidatedevices are emitting light. If a decision operation in the first deviceselection is recognized as having been performed, for example, theprojection control unit 103 causes the projection imaging device 10 toproject a visual effect that is visually recognized by the user suchthat each of the display device 20, the air conditioner device 21, andthe blower device 22 is emitting light as illustrated in FIG. 19. Notethat an image or the like indicating the candidate devices, such asarrows, may be projected instead of the light emission.

Note that in a case in which the candidate devices have light emittingfunctions, the information processing apparatus 100-1 may cause thecandidate devices to emit light. If a decision operation in the firstdevice selection is recognized, for example the device control unit 104causes the communication unit 105 to transmit light emitting requestsdirected to the candidate devices. The candidate devices that havereceived the light emitting requests causes light emitting substancesthat the devices themselves have to emit light for a predeterminedperiod of time on the basis of the light emitting request. In this case,it is possible to reduce the burden of the projection processing by thecandidate devices themselves emitting light as compared with the case inwhich projection is used.

In addition, the projection control unit 103 may cause the projectionimaging device 10 to perform projection indicating locations of thecandidate devices of the selection objects when the selection objectsare displayed. Specifically, an image indicating locations of thecandidate devices may be displayed in a display region for the selectionobjects. Further, another example of the notification of the candidatedevices according to the modification example will be described withreference to FIG. 20. FIG. 20 is a diagram illustrating another exampleof the notification of the devices to be operated, which have beenselected through the first device selection, in the informationprocessing system 1 according to the fifth modification example of theembodiment.

If a decision operation in the first device selection is recognized, theprojection control unit 103 causes the projection imaging device 10 toproject selection objects that have images indicating locations of thecandidate devices. If a decision operation in the first device selectionis recognized as having been performed, for example, the projectioncontrol unit 103 causes the projection imaging device 10 to project theselection objects 31C, 32C, and 33C that have images, such as arrows,indicating the respective locations of the blower device 22, the displaydevice 20, and the air conditioner device 21, respectively, in theregion 30 as illustrated in FIG. 20.

In addition, the notification of the candidate devices may be an audiblenotification. Specifically, the device control unit 104 may controlsound outputs from a region within which the selected devices to beoperated fall. Further, another example of the notification of thecandidate devices according to the modification example will bedescribed with reference to FIG. 21. FIG. 21 is a diagram illustratingyet another example of the notification of the devices to be operated,which have been selected through the first device selection, in theinformation processing system 1 according to the fifth modificationexample of the embodiment.

If a decision operation in the first device selection is recognized, thedevice control unit 104 causes the communication unit 105 to transmit asound output request directed to the candidate devices in a case inwhich the candidate devices have sound output functions. Each of thecandidate devices that has received the sound output request outputssound as illustrated in FIG. 21 on the basis of the sound outputrequest. Note that the output sound may be simple sound, music, orvoice. In addition, the output sound may differ or the same for each ofthe devices to be operated. In addition, the sound may be output towardthe user in a case in which the devices to be operated have a functionof adding directionality to the sound outputs.

Note that although the case in which the candidate devices have thesound output function has been described above, the informationprocessing system 1 may utilize a phenomenon such as reflection of soundoutput from a separately provided sound output device to allow the userto perceive sound as if the candidate devices output the sound.

According to the fifth modification example of the embodiment, thenotification of the candidate devices includes such a display outputthat the candidate devices or the surroundings of the candidate devicesdistinctively appear as described above. Therefore, it is possible forthe user to grasp the devices to be operated, which have been selectedas the candidate devices, at a glance. Accordingly, it is possible forthe user to determine whether or not the devices to be operated havebeen selected as intended by the user at a glance and to quicklyreselect the devices to be operated even if improper selection occurs inthe first device selection.

In addition, the notification of the candidate devices includes adisplay output indicating locations of the candidate devices in theselection objects. Therefore, the user can check the devices to beoperated, which have been selected as the candidate devices, from theoperation objects. Therefore, the user can suppress occurrence ofimproper selection by checking whether selection objects are for thedesired device to be operated before the selection. In a case in which aplurality of devices to be operated of the same kind are selected ascandidate devices, for example, it may be difficult for the user to makethe determination only from the selection objects. However, it ispossible for the user to select the desired device to be operated evenin this case by the locations of the candidate devices being presentedto the user.

In addition, the notification of the candidate devices includes a soundoutput from a region within which the candidate devices fall. Therefore,it is possible for the user to audibly check the devices to be operated,which have been selected as the candidate devices. Accordingly, it ispossible to grasp the candidate devices and to improve usability even ina case in which it is not possible to direct the line of sight to thecandidate devices or the selection objects.

4. Second Embodiment of the Present Disclosure (Display of OperationObjects)

The first embodiment of the present disclosure has been described above.Next, a second embodiment of the present disclosure will be described.In the second embodiment of the present disclosure, an embodiment of theaforementioned operation object form control function of the informationprocessing system 1 will be mainly described.

4-1. Configuration of System

First, a functional configuration of the information processing system 1according to the embodiment will be described. Note that since theprojection imaging device 10 is substantially the same as that in thefirst embodiment, only an information processing apparatus 100-2 will bedescribed. In addition, description of functions that are substantiallythe same as the functions in the first embodiment from among functionsof the information processing apparatus 100-2 will also be omitted.

(Recognition Unit)

The recognition unit 101 serves as an acquisition unit and recognizes abody form of the operation subject of devices to be operated.Specifically, the recognition unit 101 recognizes a user's posture as abody form and generates body form information related to the recognizeduser's posture. Note that since functions are substantially the same asthose of the recognition unit 101 in the first embodiment, detaileddescription will be omitted.

In addition, the recognition unit 101 recognizes a predeterminedoperation of the operation subject. Specifically, the recognition unit101 recognizes a user's display instructing operation for operationobjects. For example, the display instructing operation for theoperation objects may be a touching operation, a tapping operation, apointing operation, or the like.

(Projection Control Unit)

The projection control unit 103 serves as a display control unit andcontrols display of operation objects for the devices to be operated.Specifically, the projection control unit 103 controls complication ofthe operation objects for the devices to be operated that are visuallyrecognized as if the devices to be operated were present in a realspace, on the basis of the recognized body form of the operationsubject. For example, the projection control unit 103 causes theprojection imaging device 10 to project operation objects that arevisually recognized with complication in accordance with a user'sposture recognized by the recognition unit 101. Further, display controlfor the operation objects will be described with reference to FIGS. 22and 23. FIG. 22 is a diagram illustrating an example of display controlfor the operation objects in the information processing system 1according to the embodiment. FIG. 23 is a diagram illustrating anotherexample of display control for the operation objects in the informationprocessing system 1 according to the embodiment.

First, if a predetermined motion is performed by the user, the user'sposture is recognized by the recognition unit 101. If a user's displayinstructing operation for selection objects is recognized, for example,the recognition unit 101 recognizes the user's posture. Note that theaforementioned predetermined motion may be the selecting operation inthe second device selection in the first embodiment.

The projection control unit 103 decides operation objects linked to atarget of the predetermined motion as a display target. Specifically,the projection control unit 103 decides operation objects linked to anoperation destination of the recognized display instructing operationfor selection objects as display targets. If a tapping operation as thedisplay instruction operation for selection objects is recognized ashaving been performed to the user's thigh, for example, the projectioncontrol unit 103 decides operation objects corresponding to the user'sthigh as display targets. Note that in a case in which theaforementioned predetermined operation is the selecting operation in thesecond device selection in the first embodiment, the operation objectsthat are decided as the display targets may be operation objects for theoperation target device.

In addition, the projection control unit 103 decides complication of theoperation objects in accordance with the recognized user's posture. Thecomplication of the operation objects includes complication of displayor complications of operations. In a case in which the upper body of theuser seated as illustrated in the right diagram in FIG. 22 is inclinedbackward to a predetermined extent (in a so-called leaning-backwardstate), for example, the projection control unit 103 reduces the amountof display of the operation objects or selects an operation object thatcan be easily operated as compared with a case in which the user isseated while the upper body is inclined forward as illustrated in theright diagram in FIG. 23 (in a so-called leaning-forward state). In thismanner, it is possible to control the complication of the operationobjects in accordance with whether or not the user is in a relaxedstate. In other words, it is possible to control the complication of theoperation objects in accordance with a degree of user's concentration onoperations of a device to be operated. This is because the user is in arelaxed state in a case in which the user is in the leaning-backwardstate, for example, and it is considered that the user does not desireany detailed operation of a device to be operated.

In addition, the projection control unit 103 decides a location inaccordance with the user's form as a display location of the operationobjects. Specifically, the projection control unit 103 decides alocation in accordance with a predetermined motion performed by the useras the display location of the operation objects. More specifically, thedisplay location of the selection object may be the user's body. If atapping operation on the user's thigh is recognized, for example, theprojection control unit 103 decides the region on the user's thigh asillustrated in the left diagrams in FIGS. 22 and 23 as the displayregion of the operation objects.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the operation objects with the decided complicationat the decided location. In a case in which the user is in theleaning-backward state, for example, the projection control unit 103causes the projection imaging device 10 to project operation objects41A, 42A, and 43A as illustrated in the right diagram in FIG. 22 in theregion 40 on the user's thigh. The operation object 41A is an object forproviding operations of activating and ending the display device. Theoperation object 42A is an object for providing operations of changingchannels. The operation object 43A is an object for providing operationsof changing volume.

Meanwhile, in a case in which the user is in the leaning-forward state,the projection control unit 103 causes the projection imaging device 10to project operation objects 41B, 42B, and 43B as illustrated in theright diagram in FIG. 23 in the region 40 on the user's thigh. Theoperation objects 41B, 42B, and 43B provide operations that aresubstantially the same as those of the operation objects 41A, 42A, and43A, respectively. However, the operation object 42B is accompanied witha thumbnail image related to a channel at a destination after changing,for example, unlike the operation object 42A. In addition, the operationobject 43B can arbitrarily change the volume by moving a sliding bardrather than changing the volume one by one in a stepwise manner unlikethe operation object 43A.

4-2. Processing of System

Next, processing of the information processing system 1 will bedescribed with reference to FIG. 24. FIG. 24 is a flowchart conceptuallyillustrating an example of processing of the information processingsystem 1 according to the embodiment. Note that description ofprocessing that is substantially the same as the processing in the firstembodiment will be omitted.

The information processing system 1 acquires body form information (StepS401) and decides complication of operation objects on the basis of thebody form information (Step S402). Specifically, if a user'spredetermined motion is recognized by the recognition unit 101, theprojection control unit 103 decides operation objects projected from atarget of the predetermined motion. Then, the projection control unit103 decides complication of display or operations of the decidedoperation objects on the basis of the body form at the timing at whichthe predetermined motion is recognized.

In addition, the information processing system 1 decides a displaylocation of the operation objects on the basis of the body forminformation (Step S403). Specifically, the projection control unit 103decides a location in accordance with the user's predetermined motionrecognized by the recognition unit 101 as the display location of theoperation objects.

Next, the information processing system 1 displays the operation objectswith the decided complication at the decided location (Step S404).Specifically, the projection control unit 103 causes the projectionimaging device 10 to project the selection objects with the decidedcomplication at the decided location.

4-3. Summary of Second Embodiment

According to the second embodiment of the present disclosure, theinformation processing system 1, that is, the information processingapparatus 100-2 obtains information related to the body form of theoperation subject of the devices to be operated and controlscomplication of the operation objects for the devices to be operatedthat are visually recognized as if the devices to be operated werepresent in the real space, on the basis of the information related tothe body form as described above.

Conventionally, virtual objects fixed irrespective of user's conditionsare displayed. Therefore, there is a concern that displayed virtualobjects differ from virtual objects that the user desires. If the samevirtual objects are displayed both in a case in which the user desiresdetailed operations and in a case in which the user desires roughoperations, for example, operability may be degraded, or an operationburden may increase.

Meanwhile, according to the embodiment, it is possible to display theoperation object with complication in accordance with user's conditionsthat are estimated from the user's body form. Therefore, it is possibleto enhance possibilities that operation objects suitable for anoperation that the user desires are respectively displayed in eachuser's condition. Accordingly, it is possible to suppress variations inaccordance with user's conditions in relation to user's satisfactionabout the operation objects.

In addition, the aforementioned body form includes a posture of theaforementioned operation subject, and the aforementioned operationobjects are displayed such that the operation objects are visuallyrecognized with complication in accordance with information related tothe posture of the operation subject. A degree of user's concentrationon an operation is generally reflected to the user's posture. Therefore,it is possible to improve operability of the operation objects by thecomplication of the operation objects being controlled in accordancewith the degree of concentration on the operation estimated from theuser's posture.

In addition, the aforementioned operation objects are displayed suchthat the operation objects are visually recognized on the body of theaforementioned operation subject. Therefore, it is possible to projectthe operation objects at a location at which the user can easily operatethe operation objects even in a case in which a projection space for theoperation objects is not secured in the surroundings of the user, by theoperation objects being projected on the user's body. In addition, in acase in which the projection location is tracked, it is possible tocontinuously operate the projected operation objects since the operationobjects move even if the user moves.

In addition, the aforementioned operation objects are displayed at alocation in accordance with the body form of the aforementionedoperation subject. Therefore, it is possible to improve operability ofthe operation objects as compared with a case in which the operationobjects are fixedly displayed at a predetermined location, by thedisplay location of the operation objects being controlled in accordancewith the user's form.

In addition, the body form of the aforementioned operation subjectincludes a predetermined motion of the aforementioned operation subject,and the aforementioned operation objects are displayed at a location inaccordance with the predetermined motion. Therefore, it is possible todisplay the operation objects at a location as the user intends.Accordingly, it is possible to improve usability for the display of theoperation objects.

In addition, the aforementioned operation objects include operationobjects linked to a target of the aforementioned predetermined motion.Therefore, the user can select an operation object for the device to beoperated that the user desires to operate, by selecting the target ofthe predetermined motion. Accordingly, it is possible to omit processesrelated to the display of the operation objects as compared with a casein which a device to be operated is selected on the basis of anadditional operation.

4-4. Modification Examples

The second embodiment of the present disclosure has been describedabove. Note that the embodiment is not limited to the aforementionedexample. Hereinafter, first to ninth modification examples of theembodiment will be described.

First Modification Example

In a first modification example of the embodiment, the informationprocessing system 1 may control complication of the operation objects onthe basis of information related to another body form. Specifically, theinformation processing apparatus 100-2 causes the projection imagingdevice 10 to project the operation objects with complication inaccordance with biological information of the operation subject. Forexample, the biological information includes information related touser's pulse, body temperature, sweating, bran waves, and the like.Further, the modification example will be described in detail withreference to FIG. 25. FIG. 25 is a diagram illustrating an example ofdisplay control for operation objects in the information processingsystem 1 according to the first modification example of the embodiment.

If an operation object display instructing operation is performed, theprojection control unit 103 acquires biological information. If user'soperation object display instructing operation is recognized by therecognition unit 101, for example, the projection control unit 103acquires information related to a pulse and a body temperature as theuser's biological information. The biological information is acquiredfrom an external device 80 that is worn by the user as illustrated inFIG. 25 and generates biological information via the communication unit105. Note that the biological information may be acquired atpredetermined time intervals.

Next, the projection control unit 103 causes the projection imagingdevice 10 to project operation objects with complication decided on thebasis of the biological information. For example, the projection controlunit 103 causes the projection imaging device 10 to display theoperation objects with complication based on whether or not the acquiredinformation related to the pulse and the body temperature is equal to orgreater than threshold values. In detail, the user is regarded as beingin a relaxed state in a case in which both the pulse and the bodytemperature are less than the threshold values, operation objects 41A to43A with simple display content and operation functions as illustratedin the left diagram in FIG. 25 are projected. On the other hand, in acase in which at least one of the pulse or the body temperature is equalto or greater than the threshold value, the user is regarded as being ina nervous state, operation objects 41B to 43B with complicated displaycontent and operation functions are projected as illustrated in theright diagram in FIG. 25.

In addition, the information processing apparatus 100-2 causes theprojection imaging device 10 to project the operation objects withcomplication in accordance with information related to an action of theoperation subject (hereinafter, also referred to as action information).For example, the user's action includes an action that is notaccompanied by displacement, such as walking, cooking or reading, and anaction that is accompanied by displacement, such as cycling ortransporting. Further, another example of the modification example willbe described in detail with reference to FIG. 26. FIG. 26 is a diagramillustrating another example of display control for the operationobjects in the information processing system 1 according to the firstmodification example of the embodiment.

If an operation object display instructing operation is performed, theprojection control unit 103 acquires action information. If user'soperation object display instructing operation is recognized by therecognition unit 101, the projection control unit 103 acquires actioninformation related to user's transporting as illustrated in FIG. 26that has been recognized by the recognition unit 101, for example.

Next, the projection control unit 103 causes the projection imagingdevice 10 to project the operation objects with complication based onthe action information. If action information related to thetransporting is acquired, for example, the projection control unit 103causes the projection imaging device 10 to project an operation object44 indicating that an operation means as illustrated in FIG. 26 isvoice. In this case, the complication of the operation becomes simplerthan a manual input, and the user can operate the devices to be operatedeven when the user is transporting something and his/her hands are full.

According to the first modification example of the embodiment, theinformation related to the body form of the operation subject includesbiological information of the operation subject, and the operationobjects are displayed such that the operation objects are visuallyrecognized with complication in accordance with the biologicalinformation of the operation subject as described above. Therefore, itis possible to decide the complication of the operation objects inaccordance with whether the user is in the relaxed state or in thenervous state, in other words, in accordance with a degree ofconcentration on operations. Therefore, it is possible to reduce afeeling of strangeness, stress, or the like of the user against theoperation objects by the complication of the operation object beingadjusted in accordance with the user's state.

In addition, the body form of the operation subject includes an actionof the operation subject, and the operation objects are displayed suchthat the operation objects are visually recognized with complication inaccordance with information related to the action of the operationsubject. Therefore, it is possible for the user to smoothly operate theoperation objects without any interruption of the action by theoperation objects being displayed with the complication suitable for theuser's action.

Second Modification Example

In a second modification example of the embodiment, the informationprocessing system 1 may control complication of the operation objects onthe basis of information that is different from the body forminformation. Specifically, the information processing apparatus 100-2controls complication of the operation objects on the basis ofinformation from which the operation subject is specified. For example,the projection control unit 103 controls the complication of theoperation objects on the basis of the information from which the user isspecified (hereinafter, also referred to as user specificationinformation). Further, the modification example will be described indetail with reference to FIG. 27. FIG. 27 is a diagram illustrating anexample of display control for operation objects in the informationprocessing system 1 according to the second modification example of theembodiment.

If an operation object display instructing operation is performed, theprojection control unit 103 acquires user specification information. Ifa user's operation object display instructing operation is recognized bythe recognition unit 101, for example, the projection control unit 103acquires face recognition information as the user specificationinformation of the user. The face recognition information is obtainedthrough face recognition processing of the recognition unit 101 based onimage information related to an image in which the user appears. Forexample, users U1 and U2 as illustrated in FIG. 27 are recognized, andface recognition information related to the users UI and U2 is acquired.Note that the user specification information may be information such asan ID (Identifier) or a name instead of the face recognitioninformation.

Next, the projection control unit 103 causes the projection imagingdevice 10 to project operation objects with complication decided on thebasis of the user specification information. For example, the projectioncontrol unit 103 causes the projection imaging device 10 to display theoperation objects with complication corresponding to the acquired userspecification information. In detail, for the user U1, operation objects41A to 43A with complication associated with the user with the face thatcoincides with the face in the face recognition information of the userUI, for example, with simple display content and operation functions asillustrated in the left diagram in FIG. 27 are projected. Meanwhile, forthe user U2, operation objects 41B to 43 b with complication associatedwith the user with the face that coincides with the face in the facerecognition information of the user U2, for example, with complicateddisplay content and operation functions as illustrated in the rightdiagram in FIG. 27 are projected.

In addition, the information processing apparatus 100-2 may controlcomplication of the operation objects on the basis of information fromwhich an attribute of the operation subject is specified. Specifically,the projection control unit 103 controls the complication of theoperation objects on the basis of information from which an attribute ofthe user is specified (hereinafter, also referred to as user attributeinformation). Further, the modification example will be described indetail with reference to FIG. 28. FIG. 28 is a diagram illustratinganother example of display control for the operation objects in theinformation processing system 1 according to the second modificationexample of the embodiment.

When an operation object display instructing operation is performed, theprojection control unit 103 acquires user attribute information. If theuser's operation object display instructing operation is recognized bythe recognition unit 101, for example, the projection control unit 103acquires the user attribute information of the user. The user attributeinformation is obtained through attribute recognition processing of therecognition unit 101 based on image information related to an image inwhich the user appears. The user attribute includes an age, a sex, anationality, a race, a dominant hand, or the like.

Next, the projection control unit 103 causes the projection imagingdevice 10 to project the operation objects with complication decided onthe basis of the user attribute information. For example, the projectioncontrol unit 103 causes the projection imaging device 10 to display theoperation objects with complication corresponding to the acquired userattribute information. In detail, in a case in which a dominant handindicated by the user attribute information is the left hand, operationobjects 41C to 43C, which are arranged in a laterally inverted mannerwith respect to the operation objects 41A to 43A as illustrated in theleft diagram in FIG. 27, are projected as illustrated in FIG. 28.

Note that display or operations of at least a part of the operationobjects may be inactivated on the basis of the user specificationinformation or the user attribute information. In a case in which theage of the user indicated by the user specification information belongsto infancy, for example, the projection control unit 103 may not displayat least a part of operation functions targeted to adults or mayinactivate operations of the part in the operation objects for thedisplay device.

According to the second modification example of the embodiment, theinformation processing apparatus 100-2 controls complication of theoperation objects on the basis of the information from which theoperation subject is specified as described above. Therefore, it ispossible to set the complication of the displayed operation objects tocomplication suitable for individual users. Accordingly, it is possibleto further improve operability or usability for the individual users.

In addition, the information processing apparatus 100-2 controlscomplication of the operation objects on the basis of information fromwhich attribute of the operation subject is specified. Therefore, it ispossible to improve operability or usability as compared with a case inwhich the complication is uniformly decided, by the complication of theoperation objects being controlled in accordance with user'scharacteristics. In addition, it is possible to suppress a concern thatsafety of information deteriorates due to control of the complication ofthe operation objects without any information from which the individualusers are specified.

Third Modification Example

In a third modification example of the embodiment, the informationprocessing system 1 may control complication of the operation objects onthe basis of information other than information related to the operationsubject. Specifically, the information processing apparatus 100-2controls the complication of the operation objects on the basis ofinformation related to locations at which the operation objects arevisually recognized. For example, the projection control unit 103controls the complication of the operation objects on the basis ofinformation related to the locations at which the operation objects aredisplayed (hereinafter, also referred to as display locationinformation). Further, the modification example will be described indetail with reference to FIGS. 29 and 30. FIG. 29 is a diagramillustrating an example of display control for the operation objects inthe information processing system 1 according to the third modificationexample of the embodiment. FIG. 30 is a diagram illustrating anotherexample of display control for the operation objects in the informationprocessing system 1 according to the third modification example of theembodiment.

If an operation object display instructing operation is performed, theprojection control unit 103 acquires display location information. If auser's operation object display instructing operation is recognized bythe recognition unit 101, for example, the projection control unit 103decides a location at which the display instructing operation has beenperformed as a display location of the operation objects.

Next, the projection control unit 103 causes the projection imagingdevice 10 to project the operation objects with complication decided onthe basis of the display location information. Specifically, theprojection control unit 103 causes the projection imaging device 10 toproject the operation objects with the complication decided from adecided display location form of the operation objects. The displaylocation form includes wideness of the display location, a degree ofirregularity, a color, texture, presence of a pattern, and the like. Ina case in which the display location of the operation objects is auser's palm, and an area of a projection available region is less than athreshold value, for example, a simple operation object 42D withcomplication corresponding to the area that is less than the thresholdvalue, for example, the simple operation object 42D just for inputting achannel number as illustrated in FIG. 29 is projected. Meanwhile, in acase in which the display location of the operation objects is a table,and the area of the projection available region is equal to or greaterthan the threshold value, a complicated operation object 45 withcomplication corresponding to the area that is equal to or greater thanthe threshold value, for example, the operation object 45 with whichchannels can be selected from a program table as illustrated in FIG. 30is projected on a table 60.

According to the third modification example of the embodiment, theinformation processing apparatus 100-2 controls the complication of theoperation objects on the basis of the information related to thelocation at which the operation objects are visually recognized.Therefore, it is possible to suppress a concern that visibility of theoperation objects is degraded, by the operation objects being displayedwith complication in accordance with the display location of theoperation objects. Accordingly, it is possible to improve operabilitywhile maintaining the visibility of the operation objects.

Fourth Modification Example

In a fourth modification example of the embodiment, the display locationof the operation objects may be controlled on the basis of informationsafety. Specifically, the information processing apparatus 100-2 causesthe projection imaging device 10 to project the operation objects at alocation in accordance with a degree of information safety related tooperations of the devices to be operated. For example, the projectioncontrol unit 103 decides a location in accordance with an informationsecurity level of content of operations by the operation objects as thedisplay location of the operation objects. Further, the modificationexample will be described in detail with reference to FIG. 31. FIG. 31is a diagram illustrating an example of display control for theoperation objects in the information processing system 1 according tothe fourth modification example of the embodiment.

If an operation object display instructing operation is performed, theprojection control unit 103 acquires information from which aninformation security level of content of operations by the operationobjects is specified (hereinafter, also referred to as securityinformation). If a user's operation object display instructing operationis recognized by the recognition unit 101, for example, the projectioncontrol unit 103 acquires security information for log-in information asillustrated in FIG. 31, which is input by the operation objects.

Next, the projection control unit 103 decides a display location of theoperation objects on the basis of the security information and causesthe projection imaging device 10 to project the operation objects at thedecided display location. In a case in which the security levelindicated by the acquired security information is equal to or greaterthan a predetermined level, for example, the projection control unit 103decides the user's palm as the display location of the operationobjects. Then, the projection control unit 103 causes the projectionimaging device 10 to project an operation object 42D to the user's palmas illustrated in FIG. 31.

According to the fourth modification example of the embodiment, theoperation objects are displayed such that the operation objects arevisually recognized at the location in accordance with a degree ofinformation safety related to operations of the devices to be operatedas described above. Therefore, it is possible to prevent others fromknowing the content of the operations using the operation objects.Accordingly, it is possible to protect user's privacy or to secureinformation safety.

Fifth Modification Example

In a fifth modification example of the embodiment, the aforementionedpredetermined motion as the operation object display instructingoperation may be another motion. Specifically, the informationprocessing apparatus 100-2 causes the projection imaging device 10 toproject the operation objects at a location in accordance with apredetermined motion targeted to the devices to be operated on the basisof the predetermined motion. If a first motion directed to devices to beoperated is recognized by the recognition unit 101, for example, theprojection control unit 103 decides operation objects for the devices tobe operated as projection targets. Then, if a second motion isrecognized by the recognition unit 101, the projection control unit 103causes the projection imaging device 10 to project the operation objectsat a location in accordance with the second motion. Further, themodification example will be described in detail with reference to FIG.32. FIG. 32 is a diagram illustrating an example of display control forthe operation objects in the information processing system 1 accordingto the fifth modification example of the embodiment.

If the first motion is recognized, the projection control unit 103decides the operation objects for the devices to be operated, which havebeen specified from the first motion, as projection targets. If a handgripping motion as illustrated in the upper diagram in FIG. 32 isrecognized by the recognition unit 101, for example, the deviceselection unit 102 specifies the display device 20 that is present in adirection in which the hand is directed. Then, the projection controlunit 103 decides the operation objects for the specified display device20 as projection targets.

Next, if the second motion is recognized, the projection control unit103 decides the location in accordance with the second motion as theprojection location of the operation objects and causes the projectionimaging device 10 to project the operation objects at the decidedprojection location. If a motion of opening the gripped hand asillustrated in the lower diagram in FIG. 32 is recognized by therecognition unit 101, for example, the projection control unit 103decides the palm as the projection location of the operation objects.Then, an operation object 42D is projected to the palm as illustrated inFIG. 32.

According to the fifth modification example of the embodiment, theoperation objects linked to a target of a predetermined motion relatedto the display of the operation objects as described above includeoperation object for a device to be operated as a target of thepredetermined motion as described above. Therefore, it is possible toselect the device to be operated that the user desires to operate in theoperation object display instructing operation. Accordingly, it ispossible for the user to intuitively select the device to be operatedand to improve operability.

Sixth Modification Example

In a sixth modification example of the embodiment, the operation objectsdisplayed in accordance with the aforementioned predetermined motion asthe operation object display instructing operation may be operationobjects for devices to be operated that are present in the same realspace as the real space in which a target of the predetermined motion ispresent. Specifically, if the operation object display instructingoperation is recognized, the device selection unit 102 selects devicesto be operated that are present in the real space in which an objectthat is the target of the display instructing operation is present. Theprojection control unit 103 decides operation objects of the selecteddevices to be operated as projection targets. Then, the operationobjects decided as the projection targets are projected at a location inaccordance with the projection instructing operation. Further, themodification example will be described in detail with reference to FIGS.33 to 35. FIGS. 33 to 35 are diagrams illustrating examples of displaycontrol for the operation objects in the information processing system 1according to the sixth modification example of the embodiment.

If the operation object display instructing operation is recognized, therecognition unit 101 recognizes an object as a target of the displayinstructing operation. If a tapping operation is recognized as theoperation object display instructing operation, for example, therecognition unit 101 recognizes the table 60 situated in a living roomas illustrated in FIG. 33, in which the tapping operation has beenperformed, as a tapping destination.

Next, if an object as a target of the display instructing operation isrecognized, the device selection unit 102 selects devices to be operatedthat are present in the real space as that of the object. If a tappingoperation and a tapping destination are recognized by the recognitionunit 101, for example, the device selection unit 102 specifies theliving room, in which the table 60 recognized as the tapping destinationis present, as a device selection range. Then, the device selection unit102 selects an illumination device, an air conditioner device, and adisplay device that are present in the living room.

Next, the projection control unit 103 decides operation objects for theselected devices to be operated as projection targets and decides theobject as a target of the display instructing operation as theprojection location of the operation objects. Then, the decidedoperation objects are projected at the decided projection location. Forexample, the projection control unit 103 causes respective operationobjects 46 to 48 for the illumination device, the air conditionerdevice, and the display device selected by the device selection unit 102to be respectively projected to the table 60 to which the tappingoperation has been performed, as illustrated in FIG. 33.

Next, if an operation performed to the projected operation objects isrecognized, the projection control unit 103 changes display of theoperation objects in accordance with the recognized operation. If atouching operation performed to the operation object 48 is recognized bythe recognition unit 101, for example, the projection control unit 103changes the projected operation objects to operation objects 49 relatedto the operation object 48 as illustrated in FIG. 34. The operationobject 49 may be an operation object for performing more detailedoperation to the devices to be operated for the operation object 48. Inthe example in FIG. 34, the operation object 49 is a list ofreproduceable contents.

Similarly, if an operation performed to operation objects after a changeis recognized, the projection control unit 103 changes display of theoperation objects in accordance with the recognized operation. If atouching operation performed to a part of the operation object 49 isrecognized by the recognition unit 101, for example, the projectioncontrol unit 103 changes the projected operation objects to an operationobject 50 related to the part of the operation object 49 as illustratedin FIG. 35. The operation object 50 may be an operation object forperforming more detailed operations to the part of the operation object49. In the example in FIG. 35, the operation object 50 is a group ofoperation objects with which it is possible to reproduce, stop, rewind,and send forwarding of content and adjust volume. In addition, theoperation object 50 may include information related to the content.

Further, the projection control unit 103 causes the projection imagingdevice 10 to project a proposal of an operation of the devices to beoperated in relation to an operation related to a projected firstoperation object to the user as a second operation object. Specifically,the device control unit 104 selects a device to be operated capable ofoperating in relation to the device to be operated for the firstoperation object projected through a user's operation. Then, theprojection control unit 103 causes the projection imaging device 10 toproject a second operation object that proposes, to the user, anoperation of a device to be operated, which has been selected by thedevice selection unit 102, along with the first operation object.

If a content selecting operation by the operation object 49 asillustrated in FIG. 34 is recognized, for example, the device controlunit 104 selects an illumination device and a sound output devicecapable of operating in conjunction with a display device that displaysa movie of the selected content. The projection control unit 103 causesthe selected illumination device and sound output device to operate inconjunction with the display device or generates an operation object 51for proposing how the illumination device and the sound output deviceare to be caused to operate to the user. In detail, (1) controllingbrightness and coloration of illumination in accordance with display,(2) stopping sound outputs other than sound related to the display. (3)controlling sound in accordance with time zones (for example,suppressing heavy bass sound in midnight), and the like are listed.Then, the projection control unit 103 causes the projection imagingdevice 10 to project the operation object 51 along with the operationobject 50.

In this case, it is possible to provide an operation result beyondexpectation to the user by operations in relation to an operation thatthe user intends being additionally proposed. Therefore, it is possibleto further improve usability.

In addition, operation objects for a device to be operated, which hasbeen selected on the basis of information related to an environment inthe same real space as that of the target of the display instructingoperation (hereinafter, also referred to as spatial environmentinformation), may be displayed. The environment in the real spaceincludes a temperature, a humidity, luminance, smell, noise volume, andthe like in the real space. If an operation object display instructingoperation is performed to a table, for example, the device selectionunit 102 selects at least an air conditioner device in a case in which adisplay device, the air conditioner device, and an illumination deviceare present in the same room as the table and when the room temperatureis equal to or greater than a threshold value. Then, the projectioncontrol unit 103 causes the projection imaging device 10 to projectoperation objects for the selected air conditioner device.

In addition, operation objects for a device to be operated, which hasbeen selected on the basis of information related to a form of a personwho is present in the same real space as that of the target of thedisplay instructing operation (hereinafter, also referred to as personform information), may be displayed instead of or in addition to theaforementioned spatial environment information. The form of the personincludes states in the body, such as a pulse or a body temperature, andstates, actions, and the like outside the body, such as a posture. If anoperation object display instructing operation is performed to a tableby the user, and in a case in which there is a person other than theuser in the same room as that of the table, for example, the recognitionunit 101 acquires body temperature information of the person. Then, whenthe body temperature indicated by the acquired body temperatureinformation of the person is less than a threshold value, the deviceselection unit 102 selects at least an air conditioner device. Then, theprojection control unit 103 causes the projection imaging device 10 toproject operation objects for the selected air conditioner device.

In addition, operation objects for a device to be operated, which hasbeen selected on the basis of time information related to a time inwhich the display instructing operation has been performed, may bedisplayed instead of or in addition to the aforementioned spatialenvironment information or person form information. The time informationincludes information indicating a clock time, a time zone, a date, aday, a season or schedule information, or the like. If an operationobject display instructing operation is performed to a table by theuser, for example, the recognition unit 101 acquires time information.Next, when a time zone indicated by the acquired time information is anevening time zone, the device selection unit 102 selects at least anillumination device. Then, the projection control unit 103 causes theprojection imaging device 10 to project operation objects for theselected illumination device.

Further, processing in the modification example will be described withreference to FIG. 36. FIG. 36 is a flowchart conceptually illustratingan example of processing of the information processing system 1according to the sixth modification example of the embodiment. Note thatdescription of processing that is substantially the same as theaforementioned processing will be omitted.

The information processing system 1 acquires body form information (StepS411) and decides complication of operation objects on the basis of theacquired body form information (Step S412).

Next, the information processing system 1 determines whether or not apredetermined motion has been recognized (Step S413). Specifically, theprojection control unit 103 determines whether or not an operationobject display instructing operation has been recognized by therecognition unit 101.

If it is determined that the predetermined motion has been recognized,the information processing system 1 decides a display location ofoperation objects on the basis of the predetermined motion (Step S414).Specifically, if it is determined that the operation object displayinstructing operation has been recognized, the projection control unit103 decides a location in accordance with the display instructingoperation as a projection location of the operation objects.

Next, the information processing system 1 determines whether or notspatial environment information or person form information for a spacein which a target of the predetermined motion is present has beenacquired (Step S415). Specifically, the device selection unit 102determines whether or not information related to an environment in thespace in which the target of the operation object display instructingoperation is present or information related to a form of a person who ispresent in the space has been acquired. Note that the spatialenvironment information may be acquired from a measurement device suchas a sensor that is separately provided in the information processingsystem 1, or the person form information may be acquired from therecognition unit 101.

If it is determined that the spatial environment information or theperson form information for the space in which the target of thepredetermined operation is present has been acquired, the informationprocessing system 1 selects devices to be operated on the basis of thespatial environment information or the person form information (StepS416). Specifically, the device selection unit 102 selects a device tobe operated from among the devices to be operated that are present inthe space in which the target of the aforementioned display instructingoperation is present on the basis of the acquired spatial environmentinformation or person form information.

In addition, if it is determined that both the spatial environmentinformation and the person form information for the space in which thetarget of the predetermined motion is present have not been acquired,the information processing system 1 acquires time information (StepS417). Specifically, if it is determined that both the spatialenvironment information and the person form information have not beenacquired, the device selection unit 102 acquires time informationrelated to a time at which the aforementioned display instructingoperation has been performed.

Next, the information processing system 1 selects a device to beoperated on the basis of the time information (Step S418). Specifically,the device selection unit 102 selects a device to be operated from amongthe devices to be operated that are present in the space in which thetarget of the aforementioned display instructing operation is present onthe basis of the acquired time information.

Then, the information processing system 1 displays operation objects forthe selected device to be operated at the decided location in thedecided form (Step S419). Specifically, the projection control unit 103causes the projection imaging device 10 to project the operation objectsfor the selected device to be operated in the decided form at thedecided projection location.

Note that although the example in which a single device to be operatedis selected has been described above, a set of devices to be operatedmay be selected. For example, a set of devices to be operated related toair conditioning (for example, a set of an air conditioner device and ablower device), a set of devices to be operated related to reproductionof movies (for example, a set of a display device and a sound outputdevice), or the like may be selected.

In addition, operation objects for a set of devices to be operated maybe displayed. For example, one operation object may be projected foreach set of devices to be operated related to air conditioning or eachset of devices to be operated related to reproduction of movies. Notethat if an operation object for the set of the devices to be operated isoperated, at least a part of the devices to be operated in the set iscontrolled in accordance with the operation.

According to the sixth modification example of the embodiment, theoperation objects linked to the target of the predetermined motionrelated to the display of the operation objects as described aboveinclude operation objects for the devices to be operated that arepresent in the same real space as that of the target of thepredetermined motion as described above. Therefore, the user can causethe information processing system 1 to display operation objects for thedevice to be operated that the user desires to operate withoutindividually selecting the device to be operated. Accordingly, it ispossible to reduce efforts and time required until the device to beoperated is operated, by the operation procedure being reduced.

In addition, the device to be operated that is present in the same realspace as that of the target of the aforementioned predetermined motionon the displayed operation object is selected on the basis ofinformation related to the environment in the same real space as that ofthe target of the predetermined motion. Therefore, it is possible todisplay operation objects for a device to be operated for maintaining orimproving the environment of the space, for example, by the device to beoperated being selected in accordance with the environment of the spacein which the user is present. Accordingly, it is possible to improveusability by appropriate operation objects being displayed in accordancewith the state of the space.

In addition, the device to be operated is selected on the basis ofinformation related to the form of the person who is present in the samereal space as that of the target of the predetermined motion. Therefore,it is possible to display operation objects for a device to be operatedfor maintaining or improving a pleasant feeling that the person who ispresent in the space feels, for example, by the device to be operatedbeing selected in accordance with the form of the user or another personwho is present in the space in which the user is present. Therefore, itis possible to improve usability by appropriate operation objects beingdisplayed in accordance with the state of the person.

In addition, the device to be operated is selected on the basis of timeinformation. A state of a space in which persons feel pleasant generallydiffers depending on a time. Therefore, it is possible to appropriatelydisplay operation objects that the user desires to operate in a state ofa space that transitions with time, by the device to be operated relatedto the operation objects being selected on the basis of the timeinformation. Accordingly, it is possible to improve usability byappropriate operation objects being displayed in accordance with a time.

Seventh Modification Example

In a seventh modification example of the embodiment, operation objectsmay be automatically displayed. Specifically, the information processingapparatus 100-2 causes the projection imaging device 10 to project anoperation object for a notification to the operation subject(hereinafter, also referred to as a notification operation object) inresponse to reception of the notification. If the notification to theuser is received, for example, the projection control unit 103 decides aprojection location of the notification operation object for the deviceto be operated related to the notification. Then, the projection controlunit 103 causes the projection imaging device 10 to project thenotification operation object at the decided projection location.Further, the modification example will be described in detail withreference to FIG. 37. FIG. 37 is a diagram illustrating an example ofdisplay control for the notification operation object in the informationprocessing system 1 according to the seventh modification example of theembodiment.

The device control unit 104 determines whether not the notification fromthe device to be operated to the user has been received. For example,the device control unit 104 determines whether or not callinginformation from a phone call application such as an interphone, atelephone, or a smartphone or information such as a mail or a messagehas been acquired via the communication unit 105.

If it is determined that the notification from the device to be operatedto the user has been received, the projection control unit 103 decidesthe projection location of the notification operation object on thebasis of the location of the user as the notification destination. If itis determined that calling information has been acquired, for example,the projection control unit 103 decides the projection location of thenotification operation object for the device to be operated related tothe calling information (hereinafter, also simply referred to as anotification operation object related to the calling information) on thebasis of the location of the user who is estimated as a destination ofthe calling information.

Next, the projection control unit 103 causes the projection imagingdevice 10 to project the notification operation object at the decidedprojection location. In a case in which operation objects 46 to 48 areprojected in the vicinity of the user as illustrated in the upperdiagram in FIG. 37, for example, the projection control unit 103 causesthe projection imaging device 10 to project a notification operationobject 52 such that the notification operation object 52 is adjacent tothe operation objects 46 to 48. Note that the projection location of thenotification operation object may be decided in accordance with the useror a user's attribute. For example, the notification operation objectmay be projected on the side of a dominant hand of the user within arange that is adjacent to the operation objects.

Further, if an operation on the projected notification operation objectis recognized, the projection control unit 103 changes the form of thenotification operation object. If a touching operation on thenotification operation object 52 is recognized by the recognition unit101, for example, the projection control unit 103 switches thenotification operation object 52 to a notification operation object 53.The notification operation object 53 after the switching may be anotification operation object for operating the device to be operatedrelated to the notification operation object 52 in detail. In detail,the notification operation object 53 has three sub objects and a displayscreen as illustrated in the lower diagram in FIG. 37. The sub objectsmay be operation objects for selecting whether or not to answer callingfrom an interphone or a telephone or the like. In addition, an image inwhich a person who is the origin of the calling appears or the likeappears may be displayed on the display screen. Note that the image inwhich the person who is the origin of the calling or the like appearsmay also be displayed as illustrated in the upper diagram in FIG. 37 onthe notification operation object 52 before the switching.

An example of the processing according to the seventh modificationexample has been described above. Next, an example in which a pluralityof users are estimated to be destinations of the calling informationwill be described. Specifically, the notification operation object isdisplayed at a location at which the notification operation object isvisually recognized by each of the plurality of users. If the callinginformation is received, for example, the projection control unit 103specifies users who are estimated as destinations of the callinginformation. In the case in which the plurality of users are specified,the projection control unit 103 causes the projection imaging device 10to project the notification operation object related to the callinginformation at a location at which the notification operation object isvisually recognized by each of the plurality of specified users.Further, an example of display control for the notification operationobject for a plurality of users will be described with reference to FIG.38. FIG. 38 is a diagram illustrating an example of display control forthe notification operation object for the plurality of users in theinformation processing system 1 according to the seventh modificationexample of the embodiment.

If the notification from the device to be operated to the user isreceived, the projection control unit 103 specifies the user as thenotification destination. If calling information is received from aninterphone or the like for example, the projection control unit 103specifies users U3 to U5 as illustrated in FIG. 38 on the basis of aresult of recognizing the user by the recognition unit 101. Note thatany users in any individual spaces are specified as long as the usersare located in a space within a predetermined range (for example, in abuilding). For example, a user U3 who is located in a space (forexample, in a room) that is different from that of users U4 and U5 whilebeing located in the same building, as illustrated in FIG. 38, is alsospecified.

Next, in a case in which the plurality of users are specified, theprojection control unit 103 decides the respective projection locationsof the notification operation objects on the basis of the plurality ofrespective specified users' positions. For example, the projectioncontrol unit 103 respectively decides locations that are adjacent to therespective positions of the specified users U3 to U5 as projectionlocations of the notification operation objects.

Then, the projection control unit 103 causes the projection imagingdevice 10 to respectively project the notification operation objects atthe decided projection locations. For example, the respectivenotification operation objects 52A to 52C are projected at locationsthat are adjacent to the respective users U3 to U5 as illustrated inFIG. 38.

Note that although the example in which the notification operationobjects are respectively displayed for each of the plurality of usershas been described above, a common notification object for the pluralityof users may be displayed.

Specifically, in a case in which a plurality of users are located in thesame space, one common notification operation object for the pluralityof users may be displayed. For example, one notification operationobject may be projected for the users U4 and U5 who are located in thesame space as illustrated In FIG. 38. In that case, the notificationoperation object may be projected at a location at which thenotification operation object is visually recognized by both the usersU4 and U5, that is, a location in eyesight of both the users U4 and U5.

The example in which the notification operation object is displayed forthe plurality of users has been described above. Next, an example inwhich a notification operation object is projected such that thenotification operation object is visually recognized only by a part of aplurality of users will be described. Specifically, the notificationoperation object is displayed at a location at which the notificationoperation object is visually recognized only by specific operationsubjects. If calling information is received, for example, theprojection control unit 103 specifies users who are estimated to bedestinations of the calling information. In a case in which a singleuser is specified, the projection control unit 103 causes the projectionimaging device 10 to project the notification operation object relatedto the calling information at the specified location at which thenotification operation object is visually recognized only by thespecified users. Further, an example of display control for thenotification operation object, in which the notification operationobject is visually recognized only by a specific user, will be describedwith reference to FIG. 39. FIG. 39 is a diagram illustrating an exampleof display control for the notification operation object, in which thenotification operation object is visually recognized only by a specificuser, in the information processing system 1 according to the seventhmodification example of the embodiment.

If a notification from the device to be operated to the user isreceived, the projection control unit 103 specifies a user as anotification destination. If calling information is received from aphone calling application or the like, for example, the projectioncontrol unit 103 acquires destination information (for example, a phonenumber) from the calling information. Then, the projection control unit103 specifies a user linked to the acquired phone number. In detail, auser U8 as illustrated in FIG. 39 is specified on the basis of a resultof face recognition processing using face information acquired throughrecognition performed by the recognition unit 101 and face informationspecified from a phone number.

Next, in a case in which a single user is specified (hereinafter, alsoreferred to as a specific user) and other persons are located in thespace in which the specific user is located, the projection control unit103 decides a projection location of the notification operation objecton the basis of each of the positions of the specific user and otherpersons. For example, the projection control unit 103 acquiresinformation related to eye sight of the specific user U8 obtainedthrough recognition performed by the recognition unit 101 and of otherpersons U6 and U7 other than the specific user U8. Then, the projectioncontrol unit 103 decides a location that is in the eyesight of thespecific user U8 and is not in the eyesight of other persons U6 and U7as a projection location of the notification operation object on thebasis of information related to the eyesight. Note that the projectionlocation of the notification operation object may be body parts such ason the backs of other persons.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the notification operation object at the decidedprojection location. For example, the notification operation object 52is projected at a location, which is at dead angles from other personsU6 and U7 as illustrated in FIG. 39, at which the notification operationobject 52 is visually recognized by the specific user U8.

Note that the example in which the single user is specified has beendescribed above, specific users may be a plurality of users whosedestination information is specified. In that case, notificationoperation objects may be individually displayed for each of theplurality of specific users, or a common notification operation objectmay be displayed for the plurality of specific users.

The case in which the notification operation objects are projected suchthat the notification operation objects are visually recognized only bya part of the plurality of users has been described above. Next, anexample in which the notification operation objects are displayed onlyin a case in which there are no other persons present except forspecific users will be described. Specifically, the notificationoperation objects are displayed in a case in which there are no personsother than specific operation subjects in a space in which the specificoperation subjects are present. If calling information is received, forexample, the projection control unit 103 specifies users who areestimated to be destinations of the calling information. In a case inwhich there are persons other than the specific users in the space inwhich the specific users are present, the projection control unit 103causes the projection imaging device 10 to wait for display of thenotification operation objects until other persons move to the outsideof the space. Then, if other persons move to the outside of the space,the projection control unit 103 causes the projection imaging device 10to project the notification operation objects related to the callinginformation at locations that are adjacent to the specific users.Further, another example of display control for the notificationoperation objects, in which the notification operation objects arevisually recognized only by specific users, in the informationprocessing system 1 according to the seventh modification example of theembodiment will be described with reference to FIG. 40. FIG. 40 is adiagram illustrating another example of display control for anotification operation object that is visually recognized only the aspecific user in the information processing system 1 according to theseventh modification example of the embodiment.

If a notification from a device to be operated to a user is received,the projection control unit 103 specifies the user as a notificationdestination. For example, a user U10 as illustrated in FIG. 40 isspecified on the basis of a result of face recognition processing usingface information obtained through recognition performed by therecognition unit 101 and face information specified from destinationinformation related to the notification.

Next, in a case in which other persons are located in the space in whichthe specific user is located, the projection control unit 103 waits forother persons moving to the outside of the space. For example, theprojection control unit 103 grasps presence of the person U9 other thanthe specific user U10 from information obtained through the recognitionperformed by the recognition unit 101. Then, the projection control unit103 determines whether or not this another person U9 has moved to theoutside of the room in which the specific user U10 is located.

In addition, if this another person moves to the outside of the space,then the projection control unit 103 causes the projection imagingdevice 10 to project the notification operation object at a locationthat is adjacent to the specific user. If this another person U9 movesto the outside of the room in which the specific user U10 is located asillustrated in FIG. 40, for example, the projection control unit 103causes the projection imaging device 10 to project the notificationoperation object 52 related to the calling information at a location atwhich the notification operation object 52 is visually recognized by thespecific user U10.

According to the seventh modification example of the embodiment, theoperation objects includes the notification operation object in relationto the notification to the operation subject, and the notificationoperation object is displayed in accordance with reception of thenotification as described above. Therefore, it is possible to let theuser know about the notification by the operation objects for thedevices to be operated related to the notification being automaticallydisplayed in response to the reception. Accordingly, it is possible toreduce the time required for checking or responding to the notification.

In addition, the aforementioned notification to the operation subjectincludes a notification to a plurality of operation subjects, and theaforementioned notification operation object is displayed at a locationat which the notification operation object is visually recognized byeach of the plurality of operation subjects. Therefore, it is possibleto let each of a plurality of users notice the notification, and it ispossible for each of the plurality of users to respectively performoperations of the devices to be operated in response to thenotification. Accordingly, it is possible for another user to addressthe notification even in a case in which in which a certain user who hasnoticed the notification cannot operate the notification operationobject.

In addition, the aforementioned notification to the operation subjectincludes a notification to a specific operation subject, and theaforementioned notification operation object is displayed at a locationat which the notification operation object is visually recognized onlyby the specific operation subject. Therefore, it is possible to preventpersons other than the specific user from knowing private information ina case in which content of the notification has the private informationof the specific user. Accordingly, it is possible to balance betweenprotection of privacy for the specific user and responsiveness to thenotification.

In addition, the aforementioned notification operation object isdisplayed in a case in which persons other than the specific operationsubject are not present in a space in which the aforementioned specificoperation subject is present. Therefore, it is possible to reliablyprevent private information of the specific user from being known byother persons.

Eighth Modification Example

In an eighth modification example of the embodiment, operation objectsmay be displayed at a location in accordance with a body state of theoperation subject instead of the location in accordance with thepredetermined motion. Specifically, if a user's body state is recognizedby the recognition unit 101, the projection control unit 103 decides aprojection location in accordance with the user-s body state from theuser's body or the surroundings of the user. Further, the modificationexample will be described in detail with reference to FIG. 41. FIG. 41is a diagram illustrating an example of display control for operationobjects in the information processing system 1 according to the eighthmodification example of the embodiment.

If a predetermined motion, that is, an operation object displayinstructing operation is recognized, the recognition unit 101 recognizesthe user's body form. In a case in which the aforementioned user'sdisplay instructing operation is recognized, for example, therecognition unit 101 recognizes a state of a body part that is used inthe user's s operation (hereinafter, also referred to as an operatingpart). The operating part includes a finger, a hand, an arm, a leg, andthe like.

Next, the projection control unit 103 decides a location in accordancewith the recognized user's body state as a projection location.Specifically, the projection control unit 103 decides the projectionlocation of the operation objects on the basis of the state of theuser-s operating part recognized by the recognition unit 101. In a casein which the user holds an object with his/her left hand as illustratedin FIG. 41, for example, the projection control unit 103 determines thatthe left hand is in an operation unavailable state while the right handis in an operation available state. Then, the projection control unit103 decides the projection location of the operation objects withreference to the right hand that is in the operation available state. Inthe example in FIG. 41, a range that the right hand can reach has beendecided as a projection region 40. Then, the operation objects areprojected in the projection region 40.

In addition, the operation objects may be displayed at a location inaccordance with a posture of the operation subject. Specifically, if theuser's posture is recognized by the recognition unit 101, the projectioncontrol unit 103 decides the projection location in accordance with theuser's posture from the user's body or the surroundings of the user. Ina case in which the user is on his/her side with the left side ofhis/her own body positioned above, for example, a predetermined regionon the right side of the user, in other words, on the side to which theuser's face is directed is decided as the projection location of theoperation objects.

According to the eighth modification example of the embodiment, theoperation objects are displayed at the location in accordance with thebody state of the operation subject as described above. Therefore, it ispossible to display the operation objects at a location at which theuser can easily operate the operation objects in accordance with theuser's body state. Accordingly, it is possible to improve operability inmore use cases.

In addition, the operation objects are displayed at the location inaccordance with the posture of the operation subject. Therefore, it ispossible to display the operation objects at a location at which theoperation objects are easily recognized by the user. Accordingly, it ispossible to improve visibility of the operation objects and thus toimprove operability.

Ninth Modification Example

In a ninth modification example of the embodiment, the operation objectsmay be displayed at a location in accordance with information with whichthe operation subject is specified. Specifically, if an operation objectdisplay instructing operation is recognized, the projection control unit103 decides the display location of the operation objects on the basisof user specification information. Further, the modification examplewill be described in detail with reference to FIG. 42. FIG. 42 is adiagram illustrating an example of display control for operation objectsin the information processing system 1 according to the ninthmodification example of the embodiment. Note that description ofprocessing that is substantially the same as the aforementionedprocessing will be omitted.

If the operation object display instructing operation is recognized, theprojection control unit 103 acquires the user specification information.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the operation objects at a projection locationdecided on the basis of the user specification information.Specifically, the projection control unit 103 acquires projectionlocation information for the operation objects set for the user relatedto the acquired user specification information and causes the projectionimaging device 10 to project the operation objects at the projectionlocation that the acquired projection location information indicates. Ina case in which the right side of the user is set as the projectionlocation for the user as illustrated in FIG. 42, for example, apredetermined region on a wall on the right side of the user is decidedas the projection region 40 of the operation objects as illustrated inFIG. 42. Then, the operation objects are projected in the decidedprojection region 40.

In addition, the operation objects may be displayed at a location inaccordance with information with which attributes of the operationsubject is specified. Specifically, if an operation object displayinstructing operation is recognized, the projection control unit 103decides the display location of the operation objects on the basis ofuser's attribute information. More specifically, the projection controlunit 103 acquires dominant hand information from among attributesrelated to the user attribute information acquired when the operationobject display instructing operation is recognized and causes theprojection imaging device 10 to project the operation objects on theside of the dominant hand that the acquired dominant hand informationindicates. In a case in which the dominant hand of the user asillustrated in FIG. 42 is the right hand, for example, a predeterminedregion on a wall on the right side of the user is decided as theprojection region 40 of the operation objects as illustrated in FIG. 42.Then, the operation objects are projected in the decided projectionregion 40.

According to the ninth modification example of the embodiment, theoperation objects are displayed at the location in accordance with theinformation with which the operation subject is specified as describedabove. Therefore, it is possible to display the operation objects at alocation suitable for the individual users. Accordingly, it is possibleto further improve operability or usability of the individual users.

In addition, the operation objects are displayed at the location inaccordance with the information with which the attributes of theoperation subject are specified. Therefore, it is possible to improveoperability or usability as compared with a case in which the displaylocation is uniformly decided, by displaying the operation objects atthe location in accordance with user's characteristics. In addition, itis possible to suppress a concern that information safety deterioratesby the display location of the operation objects being controlledwithout information with which the individual users are specified.

Tenth Modification Example

In a tenth modification example of the embodiment, an operation objectmay be indirectly operated. Specifically, the operation object may beoperated on the basis of an operation performed to a user's body part.More specifically, the recognition unit 101 recognizes the operationperformed to the user's body part when the operation object isdisplayed. Then, the projection control unit 103 grasps an operationperformed to the operation object in accordance with the recognizedoperation performed to the body part and updates display of theoperation object in accordance with the grasped operation. In addition,the device control unit 104 controls the devices to be operated inaccordance with the grasped operation. Further, the modification examplewill be described in detail with reference to FIG. 43. FIG. 43 is adiagram illustrating an example of an operation of an operation objectin the information processing system 1 according to the tenthmodification example of the embodiment.

The recognition unit 101 attempts to recognize an operation performed toa user's body part when the operation objects are displayed. When theoperation object 49 is projected as illustrated in FIG. 43, for example,the recognition unit 101 attempts to recognize an operation of theoperation object 49 performed by the user to the user's own body part.The operation performed to the body part includes an operation oftouching fingers of one hand with fingers of the other hand and thelike.

If the operation performed to the user's body part is recognized, theprojection control unit 103 controls display of the operation object inaccordance with the recognized operation. If an operation of touchingthe thumb of the left hand with fingers of the right hand is recognized,for example, selected content is changed with the operation object 49.

In addition, if an operation performed to the user's body part isrecognized, the device control unit 104 controls the device to beoperated in relation to the displayed operation object in accordancewith the recognized operation. If an operation of touching a middlefinger of the left hand with fingers of the right hand is recognized,for example, a movie of content selected with the operation object 49 isreproduced by the display device.

Note that, operation functions may be respectively associated with eachof the fingers. For example, a power operation may be associated withthe thumb, channel-up may be associated with an index finger, andchannel-down may be associated with the middle finger.

According to the tenth modification example of the embodiment, theoperation object is operated on the basis of the operation performed tothe user's body part as described above. Therefore, the user can operatethe operation object without touching the operation object. Accordingly,the user can operate the operation object without moving or changing thebody posture, and it is possible to improve usability. In addition, theuser can intuitively operate the operation object. In a case in whichthe operation is performed with a somatic sensation without visuallyrecognizing the operation destination, in particular, the line of sightbecomes free. In this case, the user can operate the operation object,in other words, the device to be operated while continuously viewing amovie, for example.

Note that although the example in which the operation performed to thedisplayed operation object is an operation using a user's body part hasbeen described above, the device to be operated may be directly operatedthrough an operation using the user's body part without the operationobject being displayed.

5. Third Embodiment of the Present Disclosure (Displacement of OperationObjects)

The second embodiment of the present disclosure has been describedabove. Next, a third embodiment of the present disclosure will bedescribed. In the third embodiment of the present disclosure, anembodiment of the aforementioned operation object displacement functionin the information processing system 1 will be mainly described.

5-1. Configuration of System

First, a functional configuration of the information processing system 1according to the embodiment will be described. Note that only theinformation processing apparatus 100-3 will be described similarly tothe second embodiment. Also, description of functions that aresubstantially the same as those in the first or second embodiment fromamong the functions of the information processing apparatus 100-3 willalso be omitted.

(Recognition Unit)

The recognition unit 101 recognizes a predetermined operation performedto an operation object by an operation subject. Specifically, therecognition unit 101 recognizes an operation of gripping the operationobject and an operation of releasing the operation object. For example,the recognition unit 101 attempts to recognize an operation of grippingan operation object that is displayed on the hand and a subsequentoperation of opening the gripped hand.

(Projection Control Unit)

The projection control unit 103 serves a reference control unit andcontrols a reference of a location at which the operation objects aredisplayed such that the operation objects are visually recognized(hereinafter, also referred to as a reference of the display location)on the basis of the predetermined operation performed to the operationobjects. Specifically, if there is an object in a real space as areference of the display location, and a predetermined operationperformed to the displayed operation objects is recognized, theprojection control unit 103 changes the object as the reference of thedisplay location. Then, the operation object is displayed on the basisof the changed reference of the display location. In this manner,displacement of the operation objects is realized. Further, displacementof the operation objects will be described in detail with reference toFIG. 44. FIG. 44 is a diagram illustrating an example of displacementcontrol of the operation objects in the information processing system 1according to the third embodiment of the present disclosure.

First, a first operation from among predetermined operations for theoperation objects is recognized. If the user performs the firstoperation of gripping an operation object 42D as illustrated in thecenter diagram in FIG. 44 in a case in which the operation object 42D isprojected on a user's palm as illustrated in the upper diagram in FIG.44, for example, the first operation is recognized by the recognitionunit 101.

If the first operation is recognized, the projection control unit 103causes the projection imaging device 10 to change the form of theoperation object. If the aforementioned first operation is recognized,for example, the projection control unit 103 causes the projectionimaging device 10 to temporarily stop the projection of the operationobject 42D. When the state in which the hand is gripped continues asillustrated in the center diagram in FIG. 44, for example, theprojection of the operation object 42D is stopped. Note that the displayof the operation object 42D may be reduced in size instead of thedisplay of the operation object 42D being stopped. In this case, theuser can perform a subsequent second operation without losing sight ofthe operation object 42D.

Next, the second operation from among the predetermined operations forthe operation object is recognized. If the user performs the secondoperation of opening the gripped hand as illustrated in the lowerdiagram in FIG. 44, in other words, the second operation of releasingthe gripped operation object 42D toward a table, for example, the secondoperation is recognized by the recognition unit 101.

If the second operation is recognized, the projection control unit 103changes the reference of the display location for the operation objectselected through the aforementioned first operation on the basis of thelocation in accordance with the second operation. Specifically, theprojection control unit 103 changes the reference of the displaylocation for the operation object selected through the first operationthat is gripping to an object as a target of the second operation thatis releasing. If the second operation of releasing is recognized, forexample, the projection control unit 103 changes the reference of thedisplay location of the operation object 42D, to which the firstoperation of gripping has been performed, from the user's palm to thetable that is a target of the second operation.

If the reference of the display location is changed, the projectioncontrol unit 103 causes the projection imaging device 10 to project theoperation object on the basis of the reference of the display locationafter the changing. Specifically, the projection control unit 103controls the form of the operation object when the reference of thedisplay location is changed. More specifically, the projection controlunit 103 decides the form of the operation object after the changing onthe basis of information related to the reference of the displaylocation after the changing. The information related to the reference ofthe display location after the changing includes information with whichthe form of the reference of the display location after the changing isspecified. If the reference of the display location is changed to thetable, for example, the projection control unit 103 decides the form ofthe operation object 42D in accordance with a projection availableregion on the table. In detail, the projection control unit 103 decidescomplication of the operation object 42D in accordance with how broadthe projection available region is. In the example of the lower diagramin FIG. 44, an operation object 45 with expanded display content andoperation functions as compared with those of the operation object 42Dis projected since the projection available region on the table isbroader than the projection available region on the palm that is thereference of the display location before the changing.

In addition, the information related to the reference of the displaylocation after the changing includes information with which attributesof the reference of the display location after the changing arespecified. If the reference of the display location is changed to thetable, for example, the projection control unit 103 decides the form ofthe operation object 42D in accordance with the attributes of the table.In detail, in a case in which the type of the table is a dining table,the display content and the operation functions of the operation object42D are expanded.

In addition, the form of the operation object after the changing may bedecided on the basis of the information related to the reference of thedisplay location before the changing in a similar manner.

The displacement control for the operation objects has been describedabove. Further, the projection control unit 103 may cause the operationobject to follow the reference of the display location. Specifically,the projection control unit 103 changes the display location of theoperation objects in accordance with the displacement of the referenceof the display location. Followability of the operation objects relativeto the reference of the display location will be described in detailwith reference to FIG. 45. FIG. 45 is a diagram illustrating an exampleof followability of the operation objects relative to the reference ofthe display location in the information processing system 1 according tothe embodiment.

If the aforementioned second operation is recognized, the projectioncontrol unit 103 changes the reference of the display location to theobject that is the target of the second operation. If the secondoperation is recognized as having been performed in a direction towardthe table as illustrated in the upper diagram in FIG. 45, for example,the projection control unit 103 changes the reference of the displaylocation to the table. As a result, the operation object 45 is projectedto the table.

Then, the projection control unit 103 changes the display location ofthe operation object in accordance with the displacement of the objectthat is the reference of the display location. If the table is displacedas illustrated in the lower drawing in FIG. 45, for example, theprojection control unit 103 changes the projection location of theoperation object 45 in accordance with the displacement of the table. Asa result, it is possible to prompt the user to visually recognize thatthe operation object 45 is displaced along with the displacement of thetable as illustrated in the lower drawing in FIG. 45, that is, as if theoperation object 45 followed the displacement of the table.

Note that the object as the reference of the display location may be aperson. Specifically, the projection control unit 103 changes thedisplay location of the operation object in accordance with displacementof the person who is the reference of the display location. The case inwhich the reference of the display location is a person will bedescribed in detail with reference to FIG. 46. FIG. 46 is a diagramillustrating an example of followability of the operation object in acase in which the reference of the display location is a person in theinformation processing system 1 according to the embodiment.

First, the projection control unit 103 causes the projection imagingdevice 10 to project the operation object at a location in accordancewith attributes or a form of the person who is the reference of thedisplay location. In a case in which the dominant hand of the person whois the reference of the display location is the left hand, for example,the operation object 45 is projected at the location at which theoperation object 45 can be easily operated with the dominant hand of theperson as illustrated in the upper diagram in FIG. 46. In addition, in acase in which the body of the person who is the reference of the displaylocation faces the table, the operation object 45 is projected to thetable that is estimated to fall within the eyesight of the person asillustrated in the upper diagram in FIG. 46.

Then, the projection control unit 103 changes the display location ofthe operation object in accordance with the displacement of the personwho is the reference of the display location. If the person is displacedas illustrated in the lower diagram in FIG. 46, for example, theprojection control unit 103 changes the projection location of theoperation object 45 in accordance with the displacement of the person.As a result, it is possible to prompt the user to visually recognizethat the operation object 45 is displaced along with the displacement ofthe person as illustrated in the lower diagram in FIG. 46, that is, asif the operation object 45 followed the displacement of the person.

5-2. Processing of System

Next, processing of the information processing system 1 according to theembodiment will be described.

(Overall Processing)

First, overall processing of the information processing system 1according to the embodiment will be described with reference to FIG. 47.FIG. 47 is a flowchart conceptually illustrating an example of overallprocessing of the information processing system 1 according to theembodiment.

The information processing system 1 displays an operation object on thebasis of a reference of a display location (Step S501). Specifically, ifan operation object display instructing operation is performed, theprojection control unit 103 causes the projection imaging device 10 toproject the operation object on the basis of an initial reference of thedisplay location.

Next, the information processing system 1 determines whether or not theoperation performed to the operation object has been recognized (StepS502). Specifically, the projection control unit 103 determines whetheror not the operation performed to the operation object has beenrecognized by the recognition unit 101.

If it is determined that the operation performed to the operation objectis recognized, the information processing system 1 determines whether ornot the recognized operation is a predetermined operation (Step S503).Specifically, the projection control unit 103 determines whether or notthe operation recognized by the recognition unit 101 is thepredetermined operation. Note that in a case in which the predeterminedoperation is a set of a plurality of operations (for example, theaforementioned set of the first operation and the second operation), theprojection control unit 103 determines whether or not the recognizedoperation is an operation (for example, the first operation) that isperformed first from among predetermined operations.

If it is determined that the recognized operation is a predeterminedoperation, the information processing system 1 controls the reference ofthe display location of the operation object on the basis of thepredetermined operation (Step S504). Specifically, the projectioncontrol unit 103 changes the reference of the display location of theoperation object on the basis of the recognized predetermined operation.Note that details thereof will be described later.

If it is determined that the recognized operation is not a predeterminedoperation, the information processing system 1 controls the device to beoperated on the basis of the recognized operation (Step S505).Specifically, if an operation other than the predetermined operation isrecognized, the device control unit 104 controls the device to beoperated for the operation object on the basis of the recognizedoperation.

In addition, the information processing system 1 determines whether ornot the distance between the user and the operation object is equal toor greater than a threshold value for a predetermined time while nooperation performed to the operation object is recognized in Step S502(Step S506). Specifically, the projection control unit 103 determineswhether or not the period during which the distance between the user andthe operation object recognized by the recognition unit 101 is equal toor greater than the threshold value continues for a time that is equalto or greater than the predetermined time.

If it is determined that the distance between the user and the operationobject is equal to or greater than the threshold value for thepredetermined time, the information processing system 1 determineswhether or not the line of sight has deviated from the operation objectfor a predetermined time (Step S507). Specifically, if it is determinedthat the period during which the distance between the user and theoperation object is equal to or greater than the threshold valuecontinues for the time that is equal to or greater than thepredetermined time, the projection control unit 103 further determineswhether or not a period during which the user's line of sight deviatesfrom the operation object deviates from the operation object is equal toor greater than the predetermined time.

It is determined that the line of sight has deviated from the operationobject for the predetermined time, the information processing system 1ends the display of the operation object (Step S508). Specifically, ifit is determined that the period during which the user's line of sightdeviates from the operation object is equal to or greater than thepredetermined time, the projection control unit 103 causes theprojection imaging device 10 to end the projection of the operationobject. Note that a notification such as an alert may be provided to theuser before the projection of the operation object is ended. Thenotification may be projection of an image related to the alert, anoutput of sound, or the like.

(Processing of Controlling Reference of Display Location)

Next, processing of controlling a reference of a display location in theinformation processing system 1 will be described with reference to FIG.48. FIG. 48 is a flowchart conceptually illustrating an example of theprocessing of controlling the reference of the display location in theinformation processing system 1 according to the embodiment.

The information processing system 1 decides a reference of a displaylocation of operation objects on the basis of the predeterminedoperation (Step S511). Specifically, the projection control unit 103decides a target of a recognized predetermined operation as thereference of the display location. For example, a target to which theaforementioned second operation has been performed is decided as thereference of the display location.

Next, the information processing system 1 determines whether or not thereference of the display location has been changed (Step S512).Specifically, the projection control unit 103 determines whether or notthe reference of the display location after the decision is differentfrom the reference of the display location before the decision.

If it is determined that the reference of the display location has beenchanged, the information processing system 1 determines whether or notattribute information or form information of a destination afterchanging of the reference of the display location has been acquired(Step S513). Specifically, in a case in which the references of thedisplay locations before and after the decision differ from each other,the projection control unit 103 determines whether or not the attributeinformation or the form information of the destination after changing ofthe reference of the display destination has been acquired. Note that itmay be determined whether or not such information can be acquired.

In addition, the information processing system 1 determines whether ornot attribute information or form information of a change source of thereference of the display location has been acquired (Step S514).Specifically, the projection control unit 103 determines whether or notattribute information or form information of the change source of thereference of the display destination has been acquired. Note that it maybe determined whether such information can be acquired.

If it is determined that the attribute information or the forminformation of the destination after changing or the change source ofthe reference of the display location has been acquired, the informationprocessing system 1 decides the form of the operation objects on thebasis of the acquired attribute information or form information (StepS515). Specifically, the projection control unit 103 decidescomplication of the operation objects on the basis of the acquiredattribute information or form information of the destination afterchanging or the change source of the reference of the display location.

Note that the form of the operation objects may be decided on the basisof both the information of the destination after changing and theinformation of the change source of the reference of the displaylocation.

5-4. Summary of Third Embodiment

According to the third modification example of the present disclosure,the information processing system 1, that is, the information processingapparatus 100-3 controls the display of the operation objects for thedevice to be operated and controls the reference of the location atwhich the operation objects are displayed such that the operationobjects are visually recognized on the basis of predetermined operationperformed by the operation subject of the device to be operated to theoperation objects as described above.

Conventionally, a projection location itself of a virtual object isdisplaced in accordance with the user's predetermined motion. Therefore,if the user's predetermined motion is stopped, for example, thedisplacement of the virtual object is also stopped, and the projectionlocation of the virtual object is not affected by displacement or thelike of any object that is present at the projection location. However,real objects are displaced along with the displacement of the object onwhich the real object is placed. Since behaviors of the conventionalvirtual object differ from those of the actual object in this manner,there is a concern that the user's operation performed to the virtualobject becomes unsmooth.

Meanwhile, according to the embodiment, it is possible to control thedisplay location of the operation object not only in accordance with auser's direct operation on the operation object but also in accordancewith an indirect operation, conditions of the reference of the displaylocation irrelevant to user's actions, and the like by the reference ofthe display location of the operation object being controlled.Therefore, the user can arrange or displace the operation object as ifthe user handled a real object. Accordingly, it is possible to operatethe displacement location of the operation object with a sense of movingan actual object.

In addition, the aforementioned reference includes objects in a realspace, and the location at which the aforementioned operation object isdisplayed is changed in accordance with displacement of the objects.Therefore, it is possible to prompt the user to visually recognize as ifthe operation object is displaced between objects in the real space.Further, it is possible for the user to intuitively understand which ofobjects is a reference of the display location of the operation object,in other words, which of the objects is associated with the operationobject by the display location of the operation object following thedisplacement of the objects. Therefore, the user can easily checkwhether or not the operation object has been displaced to an intendedobject.

In addition, the aforementioned object includes the aforementionedoperation subject, and the aforementioned operation object is displayedsuch that the operation object is visually recognized at a location inaccordance with the attributes or the form of the operation subject.Therefore, the user can handle the operation object as if the operationobject was his/her belonging. Therefore, it is possible to improveusability. Further, it is possible to display the operation object suchthat the operation object follows the location suitable for the user'soperation. Accordingly, it is possible to maintain operability whiledisplacing the operation object.

In addition, the information processing apparatus 100-3 controls theform of the aforementioned operation object when the aforementionedreference is controlled. There is a probability that a displayenvironment or display conditions of the operation object change beforeand after the changing of the reference of the display location, thatis, before and after the displacement of the operation object. Thus, itis possible to display the operation object in the form suitable for thereference of the display location after the changing by controlling theform of the operation object when the reference of the display locationis controlled.

In addition, the form of the aforementioned operation object iscontrolled on the basis of the aforementioned information related to thereference after the control. Therefore, it is possible to increase aprobability at which the form after the change is suitable for thedisplacement destination by the form being changed in accordance withthe displacement destination of the operation object.

In addition, the aforementioned information related to the referenceafter the control includes information with which attributes or a formof the aforementioned reference after the control is specified.Therefore, it is possible to more reliably increase the probability atwhich the form after the changing is suitable for the displacementdestination by the operation object being displayed in the form inaccordance with characteristics of the displacement destination. Inaddition, it is possible to more reliably increase the probability atwhich the form after the changing is suitable for the displacementdestination irrespective of a change in conditions of the displacementdestination by the operation object being displayed in the form inaccordance with conditions of the displacement destination that canvary.

In addition, the form of the aforementioned operation object includescomplication of the operation object. Therefore, it is possible tomaintain or improve visibility or operability of the operation object bycomplication of display or operations being controlled in accordancewith the displacement of the operation object.

In addition, the aforementioned predetermined operation includes the setof the first operation and the second operation on the aforementionedoperation object, and the information processing apparatus 100-3 changesthe reference of the operation object selected through the firstoperation on the basis of a location in accordance with the secondoperation. Therefore, the user can displace only the desired operationobject by the operation object as the target of the displacement beingexplicitly selected. Accordingly, it is not necessary for any operationobject that the user does not intend to be displaced, and it is possibleto improve operability of an operation object displacement operation. Inaddition, the user can smoothly perform the operation objectdisplacement operation by the operation object being displaced by aseries of first and second operations. Accordingly, it is possible tosuppress complexity in the operation object displacement operation.

5-4. Modification Examples

The third embodiment of the present disclosure has been described above.Note that the embodiment is not limited to the aforementioned example.Hereinafter, first to sixth modification examples of the embodiment willbe described.

First Modification Example

In a first modification example of the embodiment, the predeterminedoperation may be another operation. Specifically, the predeterminedoperation includes contact between targets of references of the displaylocation. If an object as a reference of the display location isrecognized as having been brought into contact with another object bythe recognition unit 101, for example, the projection control unit 103changes the reference of the display location to this another object.Further, the modification example will be described in detail withreference to FIG. 49. FIG. 49 is a diagram illustrating an example of apredetermined operation in the information processing system 1 accordingto the first modification example of the embodiment.

The projection control unit 103 causes the projection imaging device 10to project an operation object to the object as the reference of thedisplay location. For example, an operation object 42D, the reference ofthe display location of which is a user U11 or a hand or a palm of theuser U11, is projected to the palm of the user U11 as illustrated in theupper diagram in FIG. 49.

Next, if the object as the reference of the display location isrecognized as having been brought into contact with another object, theprojection control unit 103 changes the reference of the displaylocation to this another object. If the user U11 with the hand on whichthe operation object 42D is being projected is recognized as havingshaking hands with another user U12 with that hand as illustrated in thecenter diagram in FIG. 49, for example, the projection control unit 103changes this another user U12 or the hand or the palm of the user U12 tothe reference of the display location of the operation object 42D.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the operation object on the basis of the referenceof the display location after the changing. As illustrated in the lowerdiagram in FIG. 49, for example, the operation object 42D is projectedon the palm, which is the reference of the display location after thechanging, of aforementioned another user U12. Note that the operationobject 42D is not projected for the aforementioned user U11 who is thereference of the display location before the changing. In this manner,it is possible to prompt the user to visually recognize as if theoperation object has been displaced by the user shaking hands.

The example in which the reference of the display location is changed bythe contact between the targets of the references of the displaylocation has been described above. However, the predetermined operationmay be another operation. Specifically, the predetermined operation maybe an operation of causing the operation object to rotate. For example,if the operation of rotating the operation object is recognized by therecognition unit 101, the projection control unit 103 changes thereference of the display location to an object at a rotationdestination. Further another mode of the modification example will bedescribed in detail with reference to FIG. 50. FIG. 50 is a diagramillustrating another example of the predetermined operation in theinformation processing system 1 according to the first modificationexample of the embodiment.

The projection control unit 103 causes the projection imaging device 10to project the operation object on the basis of the reference of thedisplay location. For example, the operation object 45, the reference ofthe display location of which is a user U13, is projected on a tablesuch that the operation object 45 confronts the user 13 as illustratedin the upper diagram in FIG. 50.

Next, if an operation of rotating the operation object is recognized,the projection control unit 103 changes the reference of the displaylocation to the reference of the display location for a user who ispresent in a direction related to the rotation. If the operation ofcausing the operation object 45 to rotate such that the operation object45 faces the direction of a user U14, which is performed by the userU13, is recognized as illustrated in the upper diagram in FIG. 50, forexample, the projection control unit 103 changes the reference of thedisplay location of the operation object 45 from the user U13 to theuser U14.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the operation object on the basis of the referenceof the display location after the changing. For example, the operationobject 45 is caused to rotate such that the operation object 45confronts the user U14 as illustrated in the lower diagram in FIG. 50.In this manner, it is possible to prompt the user to visually recognizeas if an owner of the operation object has been changed by the user'srotating operation.

The example in which the reference of the display location is changed bythe operation of rotating the operation object has been described above.However, the predetermined operation may be another operation.Specifically, the predetermined operation may be an operation ofsnapping the operation object to the destination after changing of thereference of the display location. If the operation of snapping theoperation object is recognized by the recognition unit 101, for example,the projection control unit 103 changes the reference of the displaylocation to the object that is estimated from the snapping operation.Further, another mode of the modification example will be described withreference to FIGS. 51 to 53. FIGS. 51 to 53 are diagrams illustratingother examples of the predetermined operation in the informationprocessing system 1 according to the first modification example of theembodiment.

First, a first example of control for the reference of the displaylocation based on the operation of snapping the operation object will bedescribed with reference to FIG. 51.

The projection control unit 103 causes the projection imaging device 10to project the operation object on the basis of the reference of thedisplay location. For example, the operation object 45, the reference ofthe display location of which is a user U15, is projected in a rangethat the hand of the user U15 reaches as illustrated in FIG. 51.

Next, if the operation of snapping the operation object is recognized,the projection control unit 103 changes the reference of the displaylocation to an object that is present in the snapping direction on thebasis of the amount of the operation. If the operation object 45 isdisplaced by the user U15 to an extend exceeding a predetermined rangewith reference to the projection location before the operation asillustrated in FIG. 51, for example, the projection control unit 103estimates the direction in which the operation object 45 has beensnapped. Then, the projection control unit 103 changes the reference ofthe display location to a user U16 who is present in the estimateddirection. Note that in a case in which the operation object 45 isdisplaced within the predetermined range, the reference of the displaylocation is not changed.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the operation object on the basis of the referenceof the display location after the changing. For example, the operationobject 45 is projected such that the operation object is displaced inthe snapping direction until the operation object 45 reaches the rangethat the hand of the user U16 who is the reference of the displaylocation after the changing reaches. Note that the operation object 45may be temporarily deleted and then be projected near the user U16instead of being projected such that the operation object 45 isdisplaced.

Next, a second example of control for the reference of the displaylocation based on the operation of snapping the operation object will bedescribed with reference to FIG. 52.

The projection control unit 103 causes the projection imaging device 10to project the operation object on the basis of the reference of thedisplay location. For example, the operation object 45, the reference ofthe display location of which is a user U17, is projected in a rangethat the hand of the user U17 reaches as illustrated in the upperdiagram and the lower diagram of FIG. 52.

Next, if the operation of snapping the operation object is recognized,the projection control unit 103 changes the reference of the displaylocation to an object that is present in the snapping direction on thebasis of a form of an operation body that performs the operation. If anoperation of snapping the operation object 45 with one finger asillustrated in the upper diagram in FIG. 52, which is performed by auser U17, is recognized, for example, the projection location of theoperation object is displaced in accordance with the operation. In thiscase, the reference of the display location is not changed. Meanwhile,if an operation of snapping the operation object 45 with five fingers asillustrated in the lower diagram in FIG. 52, which is performed by theuser U17, is recognized, the projection control unit 103 estimates thedirection in which the operation object 45 has been snapped with theoperation. Then, the projection control unit 103 changes the referenceof the display location to a user U18 who is present in the estimateddirection.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the operation object on the basis of the referenceof the display location after the changing. Since details aresubstantially the same as those in the example in FIG. 51, descriptionthereof will be omitted.

Next, a third example of control for the reference of the displaylocation based on the operation of snapping the operation object will bedescribed with reference to FIG. 53.

The projection control unit 103 causes the projection imaging device 10to project the operation object on the basis of the reference of thedisplay location. For example, the operation object 45, the reference ofthe display location of which is a user U19, is projected in a rangethat the hand of the user U19 reaches as illustrated in the upperdiagram and the lower diagram of FIG. 53.

Next, if the operation of snapping the operation object is recognized,the projection control unit 103 changes the reference of the displaylocation to an object that is present in the snapping direction on thebasis of a part of the operation object that has been touched during theoperation. If an operation of snapping the operation object 45 by a userU19 in a state in which an information display part of the operationobject 45 is touched as illustrated in the upper diagram in FIG. 53 isrecognized, for example, the projection location of the operation object45 is displaced in accordance with the operation. In this case, thereference of the display location is not changed. Meanwhile, if anoperation of snapping the operation object 45 by the user U19 in a statein which an upper end part of the operation object 45 is touched asillustrated in the lower diagram in FIG. 53 is recognized, theprojection control unit 103 estimates the direction in which theoperation object 45 has been snapped with the operation. Then, theprojection control unit 103 changes the reference of the displaylocation to a user U20 who is present in the estimated direction.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the operation object on the basis of the referenceof the display location after the changing. Since details aresubstantially the same as those in the example in FIG. 51, descriptionthereof will be omitted.

According to the first modification example of the embodiment, thepredetermined operation includes contact between the targets of thereferences of the display locations as described above. Therefore, it ispossible to prompt the user to visually recognize as if the operationobject has been displaced between the objects. Accordingly, the user canintuitively pass and receive the operation object.

In addition, the predetermined operation includes the operation ofrotating the operation object. Therefore, the user can change anoperator of the operation object by changing the orientation of theoperation object. Accordingly, the user can simply change the operatorof the operation object.

In addition, the predetermined operation includes the operation ofsnapping the operation object to the destination after changing of thereference of the display location. Therefore, the user can change theoperator of the operation object to a desired user even in a case inwhich the displacement destination of the operation object is notpresent at a close location. Accordingly, it is possible to reduce theburden on the user for the predetermined operation.

Second Modification Example

In a second modification example of the embodiment, the reference of thedisplay location may be a position in a real space. Specifically, theprojection control unit 103 causes the projection imaging device 10 toproject the operation object on the basis of the position in the realspace, which is the reference of the display location. Further, theprojection control unit 103 controls the type of the reference of thedisplay location in accordance with a form of the operation subject whenthe predetermined operation is performed. The modification example willbe described in detail with reference to FIG. 54. FIG. 54 is a diagramillustrating an example of a change in the reference of the displaylocation in the information processing system 1 according to the secondmodification example of the embodiment.

If the predetermined operation is recognized, the recognition unit 101recognizes the form of the operation subject who is performing thepredetermined operation. If an operation of opening the left hand towardthe table as illustrated in the upper diagram in FIG. 54 (theaforementioned second operation) is recognized, for example, therecognition unit 101 recognizes that the user who is performing theoperation is gripping his/her left wrist with his/her right hand.

The projection control unit 103 selects the type of the reference of thedisplay location in accordance with the form of the operation subjectwho is performing the predetermined operation. Specifically, theprojection control unit 103 selects the type of the reference of thedisplay location from an object, a person, a position, and the like inaccordance with the form of the user who is performing the predeterminedoperation, which has been recognized by the recognition unit 101. If theuser who is performing the predetermined operation is recognized asgripping his/her wrist of the hand with which the operation is beingperformed with his/her other hand, for example, the projection controlunit 103 selects the position in the real space as the target of thereference of the display location.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the operation object on the basis of the referenceof the display location after the changing. For example, the operationobject 45 is projected on a table that is arranged at the position thatis the reference of the display location after the changing. Note thatsince the reference of the display location is the position at which thetable is arranged rather than the table, the display location of theoperation object 45 is not displaced even if the table is displaced asillustrated in the lower diagram in FIG. 54, for example, and theoperation object 45 is projected on the floor.

According to the second modification example of the embodiment, thereference of the display location includes the position in the realspace, and the display location of the operation object is displayed onthe basis of the position in the real space as described above.Therefore, even if an object is arranged at the location at which theoperation object is projected, it is possible to continue the projectionof the operation object at the same position irrespective of thedisplacement of the object. Accordingly, it is possible to suppressdisplacement of the operation object despite user's intention.

In addition, the information processing apparatus 100-3 controls thetype of the reference of the display location in accordance with theform of the operation subject when the predetermined operation isperformed. Therefore, the user can designate the type of the destinationafter changing of the reference of the display location, that is, adestination to which the operation object is to be associated.Accordingly, the user can handle the operation object as the userintends, and it is possible to improve usability.

Third Modification Example

In a third modification example of the embodiment, a form of anoperation object may be controlled on the basis of information relatedto an objective entity of an operation of the operation object when thereference of the display location is controlled. Specifically, the formof the operation object may be controlled on the basis of informationrelated to a state of the operation object (hereinafter, also referredto as state information). In a case in which display content of theoperation object differs from display content in an initial state whenthe reference of the display location of the operation object ischanged, for example, the projection control unit 103 decides to returnthe display content of the operation object to the display content inthe initial state.

In addition, the form of the operation object may be controlled on thebasis of information related to a target to be operated via theoperation object (hereinafter, also referred to as operation targetinformation). For example, the projection control unit 103 decidesdisplay content of the operation object in accordance with the type ofcontent that the device to be operated, which is operated by theoperation object, reproduces when the reference of the display locationof the operation object is changed.

Further, processing in the modification example will be described indetail with reference to FIG. 55. FIG. 55 is a flowchart conceptuallyillustrating an example of processing of controlling the reference ofthe display location in the information processing system 1 according tothe third modification example of the embodiment. Note that descriptionof processing that is substantially the same as the aforementionedprocessing will be omitted.

The information processing system 1 decides a reference of a displaylocation of the operation object on the basis of a predeterminedoperation (Step S521) and determines whether or not the reference of thedisplay location has been changed (Step S522).

If it is determined that the reference of the display location has beenchanged, the information processing system 1 determines whether or notstate information of the operation object has been acquired (Step S523)and decides a form of the operation object on the basis of the stateinformation (Step S524). Specifically, if information with whichcomplication of the operation object is specified is acquired, theprojection control unit 103 determines whether or not complication ofdisplay content or operation functions of the operation object differsfrom complication in the initial state on the basis of the information.If it is determined that the complication differs from that in theinitial state, the projection control unit 103 decides to return thecomplication of the operation object to complication in the initialstate.

In addition, the information processing system 1 determines whether ornot operation target information of the operation object has beenacquired (Step S525) and decides a form of the operation object on thebasis of the operation target information (Step S526). Specifically, ifinformation from which the device to be operated or content handled bythe device to be operated, which is operated by the operation object, isspecified is acquired, the projection control unit 103 determines thetype of the device to be operated or of the content specified from theinformation. Then, the projection control unit 103 decides the form ofthe operation object in accordance with the type of the device to beoperated or of the content.

According to the third modification example of the embodiment, the formof the operation object is controlled on the basis of the informationrelated to the objective entity of the operation in relation to theoperation object as described above. In a case in which the change ofthe reference of the display location corresponds to a change inoperators, for example, and the operation object is displaced directlyto the operator after the changing, there may be a case in whichoperation content of the operator before the changing is known by theoperator after the changing. In a case in which content for adults isoperated by the operation object, in particular, it is not favorable todisplace the operation object to a child in the same form. Meanwhile,according to the modification example, it is possible to cause the formof the operation object to change in accordance with the form of theoperation object or content or the like that is operated by theoperation object when the operation object is displaced. In this manner,it is possible to suppress the operation content from being known by theoperator as the displacement destination.

Fourth Modification Example

In a fourth modification example of the embodiment, a destination afterchanging of a reference of a display location may be explicitlypresented for the user. Specifically, the projection control unit 103controls display of a display object, from which a destination afterchanging of the reference of the display location is specified. Forexample, the projection control unit 103 estimates an object as acandidate of the destination after changing of the reference of thedisplay location and causes the projection imaging device 10 to projectthe display object from which the estimated object is grasped by theuser. Further, the modification example will be described in detail withreference to FIG. 56. FIG. 56 is a diagram illustrating an example inwhich the destination after changing of the reference of the displaylocation of the operation object is explicitly represented in theinformation processing system 1 according to the fourth modificationexample of the embodiment.

The projection control unit 103 causes the projection imaging device 10to project the operation object on the basis of the reference of thedisplay location. For example, the operation object 45, the reference ofthe display location of which is a user U21, is projected in a rangethat the hand of the user U21 reaches as illustrated in FIG. 56.

Next, if a start of an operation of changing the reference of thedisplay location of the operation object is recognized, the projectioncontrol unit 103 estimates an object as a candidate of the destinationafter changing of the reference of the display location. If an operationof touching the operation object 45 with a finger is recognized, forexample, the projection control unit 103 searches for an object that canbe a reference of a display location that is present in the surroundingsof the operation object. As a result, users U22 and U23 are discovered,and the users U22 and U23 are estimated to be candidates for thedestination after changing of the reference of the display location.

Next, the projection control unit 103 causes the projection imagingdevice 10 to project the display object in which the estimatedcandidates for the destination after changing are explicitlyrepresented. For example, display objects 54A and 54B as arrows directedto the users U22 and U23 estimated as candidates for the destinationafter changing of the reference of the display location from theoperation object 45 are respectively projected as illustrated in FIG.56.

Then, the projection control unit 103 changes the reference of thedisplay location in accordance with an operation of changing thereference of the display location of the operation object. As a result,the operation object is visually recognized by the user as if theoperation object was displaced.

According to the fourth modification example of the embodiment, theinformation processing apparatus 100-3 controls display of the displayobject from which the destination after changing of the reference of thedisplay location is specified as described above. Therefore, the usercan previously grasp an object to which the operation object can bedisplaced. Accordingly, it is possible to suppress a concern thatdisplacement of the operation object to the intended object is failed.In addition, it is possible to reduce a concern that the user fails theoperation by the display object for guiding the operation direction,such as an arrow, being displayed.

Fifth Modification Example

In a fifth modification example of the embodiment, a plurality ofoperation objects may be merged. Specifically, in a case in whichanother operation object is present in relation to a destination afterchanging of a reference of a display location, the projection controlunit 103 merges the operation object, the reference of the displaylocation of which is changed, with this another operation object as thedestination after changing of the reference of the display location. Forexample, an operation object, the reference of the display location ofwhich is changed (hereinafter, also referred to as an operation objectas a change source) and an operation object as the destination afterchanging of the display location (hereinafter, also referred to as anoperation object as the destination after changing) may be integrated.Further, the modification example will be described in detail withreference to FIG. 57. FIG. 57 is a diagram illustrating an example ofmerging of operation objects in the information processing system 1according to the fifth modification example of the embodiment.

The projection control unit 103 causes the projection imaging device 10to project the operation objects on the basis of the reference of thedisplay location. For example, the operation object 45, the reference ofthe display location of which is a user U24, is projected within a rangethat the hand of the user U24 reaches as illustrated in the upperdiagram in FIG. 57. In addition, an operation object 49, the referenceof the display location is a user U24, is projected in a range that thehand of the user U25 reaches as illustrated in the upper diagram in FIG.57.

Next, if an operation of changing the reference of the display locationof the operation objects is recognized, the projection control unit 103changes the reference of the display location to an object at thedestination after changing of the changing operation. Then, theoperation objects are projected such that the operation objects aredisplaced to the object at the destination after changing. If anoperation of displacing the operation object 45 toward the user U25,which is performed by the user U24, as illustrated in the upper diagramin FIG. 57, is recognized, for example, the projection control unit 103changes the reference of the display location of the operation object 45to the user U25. Then, the operation object 45 is projected such thatthe operation object 45 is displaced toward the user U25.

Then, if the operation object reaches the operation object in relationto the destination after changing of the reference of the displaylocation, the projection control unit 103 integrates operation objects.Then, an operation object obtained through the integration is projected.If the operation object 45 is displaced to the operation object 49 forthe user U25, for example, the projection control unit 103 causes a partof the operation object 45 and a part of the operation object 49 to beintegrated with each other. Then, an operation object 55 obtainedthrough the integration is projected for the user U25 as illustrated inthe lower diagram in FIG. 57.

Note that the merging of the operation objects may be fusion of anoperation object of a change source and an operation object of adestination after changing. For example, an operation object related toa recording reservation for a recording device, such as a hard diskrecorder, may be displayed through fusion of an operation object for adisplay device and an operation object for a clock device.

In addition, one operation object may be divided into a plurality ofoperation objects. If a dividing operation performed by the user to adisplayed operation object is recognized, for example, the projectioncontrol unit 103 decides a plurality of operation objects in relation tothe operation object. Then, the decided plurality of operation objectsare displayed instead of the original operation object.

According to the fifth modification example of the embodiment, in thecase in which another operation object is present in relation to thedestination after changing of the reference of the display location, theinformation processing apparatus 100-3 merges the operation object, thereference of the display location of which is changed, and this anotheroperation object as the destination after changing of the reference ofthe display location as described above. Therefore, the user canintuitively organize the operation objects. In addition, the user canintuitively change the displayed operation object to the desiredoperation object by the operation object through which it is possible toperform an operation in relation to the operation object merged throughthe merging of the operation objects being displayed. Accordingly, it ispossible to improve usability.

Sixth Modification Example

In a sixth modification example of the embodiment, the informationprocessing apparatus 100-2 may further control attributes of anoperation object on the basis of a predetermined operation.Specifically, if a predetermined operation is recognized by therecognition unit 101, the projection control unit 103 controlsattributes of the operation object, to which the predetermined operationhas been performed, in accordance with a target of the predeterminedoperation. Attributes of the operation object includes an operationsubject who operates the operation object, for example. Further, themodification example will be described in detail with reference to FIGS.49 and 58. FIG. 58 is a diagram illustrating an example of displaycontrol for an operation object in the information processing system 1according to the sixth modification example of the embodiment.

If a predetermined operation performed to a person is recognized, theprojection control unit 103 changes a reference of a display locationand also changes attributes of the operation object. As described abovewith reference to FIG. 49, for example, if shaking hands of the user isrecognized, the reference of the display location is changed to a userwho is a counterpart of the hand shaking. At that time, an owner of theoperation object, which is one of the attributes of the operationobject, is also changed to a user who is the counterpart of the handshaking.

Next, if a predetermined operation performed to an object except for aperson is recognized, the projection control unit 103 changes only thereference of the display location. If a predetermined operationperformed by the user who is the aforementioned counterpart of the handshaking to a table is recognized as illustrated in the upper diagram inFIG. 58, for example, the projection control unit 103 changes thereference of the display location to the table. Then, the operationobject 45 is projected on the table. In this case, the owner of theoperation object is not changed. Note that other attributes of theoperation object may be changed.

Next, the projection control unit 103 controls a form of the operationobject on the basis of the attributes of the operation object.Specifically, the projection control unit 103 displaces the operationobject on the basis of the reference of the display location in responseto the displacement of the operation subject who operates the operationobject. As illustrated in the lower diagram in FIG. 58, for example, theoperation object 45 projected on the table is projected such that theoperation object 45 is displaced on the table in accordance with thedisplacement of the user who is the owner of the operation object 45.Since the reference of the display location is the table, the operationobject 45 is not projected outside the table even if the user moves awayfrom the table, in this example. Note that in the case in which thereference of the display location is the user, the operation object isdisplaced to the outside of the table in accordance with thedisplacement of the user.

Note that although the example in which the display location of theoperation object is controlled on the basis of the attributes of theoperation object has been described above, complication of the operationobject may be controlled. For example, an operation object with displaycontent or operation functions in accordance with an owner of theoperation object may be projected.

According to the sixth modification example of the embodiment, theinformation processing apparatus 100-3 further controls the attributesof the operation object on the basis of the predetermined operation asdescribed above. Therefore, it is possible to more finely control theprojection of the operation object by not only the reference of thedisplay location but also the attributes of the operation object beingcontrolled on the basis of the predetermined operation. Accordingly, itis possible to address a variety of user needs.

In addition, the aforementioned attributes of the operation objectinclude the operation subject who operates the operation object.Therefore, it is possible to more finely control the projection of theoperation object in accordance with the user of the operation object.Accordingly, it is possible to project the operation object inaccordance with the individual users.

In addition, the information processing apparatus 100-3 controls theform of the operation object on the basis of the aforementionedattributes of the operation object. Therefore, it is possible toproperly adjust visibility or operability of the operation object by theoperation object being displayed in the form in accordance with theattributes of the operation object. In a case in which an attribute ofthe operation object is users, in particular, it is possible to furtherenhance usability since the operation object in the form suitable forthe individual users is projected.

Seventh Modification Example

In a seventh modification example of the embodiment, an operation objectmay be reproduced. Specifically, the projection control unit 103reproduces the operation object on the basis of a predeterminedoperation. If the predetermined operation is recognized, for example,the projection control unit 103 causes the projection imaging device 10to project a new operation object obtained through reproduction of anoperation object related to the predetermined operation. Themodification example will be described in detail with reference to FIG.59. FIG. 59 is a diagram illustrating an example of display control foran operation object in the information processing system 1 according tothe seventh modification example of the embodiment.

The projection control unit 103 causes the projection imaging device 10to project an operation object for a device to be operated on the basisof a reference of a display location. For example, an operation object56 for a smartphone 70 as illustrated in the left diagram in FIG. 59,for example, is projected on a table that a hand of a user U26 reaches.The operation object 56 may be a music file list, for example. Further,another operation object 57A as illustrated in the left diagram in FIG.59 is newly projected by a selecting operation performed by the user U26to the operation object 56. The operation object 57A may be a musicfile, for example.

Next, if a reproduction operation performed to the projected operationobject is recognized, the projection control unit 103 causes theprojection imaging device 10 to project the new operation object on thebasis of the projected operation object. If an operation of dividing theoperation object 57A into two as illustrated in the right diagram inFIG. 59 is recognized, for example, the projection control unit 103causes the projection imaging device 10 to project an operation object57B that is substantially the same as the operation object 57A.

Further, the reference of the display location of the new operationobject obtained through the reproduction may be controlled on the basisof a predetermined operation. If an operation of passing the operationobject 57B obtained through the reproduction by the user U26 to anotheruser U27 is recognized as illustrated in the right diagram in FIG. 59,for example, the projection control unit 103 changes the reference ofthe display location of the operation object 57B to the user U27.

Note that the operation object may be displaced via communication. In acase in which the reference of the display location of the operationobject has been changed to an object that is present in anotherbuilding, for example, the projection control unit 103 may cause aprojection device in this another building to project informationrelated to the operation object via the communication. Further, thedestination after changing of the reference of the display location maybe a virtual object. For example, the user U27 as illustrated in theright diagram in FIG. 59 may be present in a building that is differentfrom a building in which the user U26 is present, and a moviecorresponding to the user U27 may be projected. Then, if an operation ofpassing the operation object 57B from the user U26 to the user U27 whois projected as the movie is recognized, the operation object 57B isprojected for the user U27 as an actual object via the communication.

The example in which the operation object is reproduced has beendescribed above. Synchronization of the operation object related to thereproduction may further be controlled. Specifically, the projectioncontrol unit 103 controls synchronization between the reproducedoperation object and the operation object as a reproduction source onthe basis of the form of the operation subject of the predeterminedoperation. The control of the synchronization of the operation objectrelated to the reproduction will be described in detail with referenceto FIG. 60. FIG. 60 is a diagram illustrating an example of displaycontrol for an operation object related to reproduction in theinformation processing system 1 according to the seventh modificationexample of the embodiment.

The projection control unit 103 causes the projection imaging device 10to project the operation object on the basis of the reference of thedisplay location. As illustrated in the upper diagram and the lowerdiagram in FIG. 60, for example, an operation object 45A is projectedwithin a range that the hand of the user U28 can reach.

Next, if a reproduction operation for the projected operation object isrecognized, the recognition unit 101 recognizes the form of thereproduction operation. If an operation of snapping the operation object45A performed by a user U28 is recognized as illustrated in the upperdiagram and the lower diagram in FIG. 60, for example, the number offingers of the user U28 with which the operation has been performed isrecognized. One finger is recognized in the upper diagram in FIG. 60,and two fingers are recognized in the lower diagram in FIG. 60.

The projection control unit 103 decides whether or not the reproducedoperation object has been synchronized with the operation object as areproduction source on the basis of the form of the reproductionoperation. For example, the projection control unit 103 decides whetheror not the synchronization has been performed in accordance with thenumber of fingers in the recognized reproduction operation.Non-synchronization of the operation object related to the reproductionis decided in the upper diagram in FIG. 60 in which one finger isrecognized while synchronization of the operation object related to thereproduction is decided in the lower diagram in FIG. 60 in which twofingers are recognized. Content of the synchronization includessynchronization of complication, such as display content or operationfunctions, for example, of the operation object.

Then, the projection control unit 103 causes the projection imagingdevice 10 to project the reproduced operation object, and the controlrelated to the operation object is performed in accordance with whetheror not the synchronization has been performed. As illustrated in theupper diagram in FIG. 60, for example, an operation object 45B that issubstantially the same as the operation object 45A is projected on auser U29 as the reference of the display location. Since the operationobjects 45A and 45B are not synchronized with each other, the operationobjects 45A and 45B are independently operated. Meanwhile, an operationobject 58 that is substantially the same as a part of the operationobject 45A is projected on the user U29 as the reference of the displaylocation as illustrated in the lower diagram in FIG. 60. Since theoperation objects 45A and 58 are synchronized with each other, thedisplay of the operation object 58 also changes if the display of theoperation object 45A changes, for example. Note that the reproduction ofthe operation object may be reproduction of only a part of the operationobject as the reproduction source as illustrated in the lower diagram inFIG. 60.

Note that although the example in which whether or not the operationobject related to the reproduction has been synchronized is controlledhas been described above, synchronization of other modes may becontrolled. For example, a synchronized part of the operation objectrelated to the reproduction may be controlled. In detail, a part of theoperation object to be synchronized, whether the part is a part orentirety of the operation object, or the like may be controlled. Inaddition, a time during which synchronization or non-synchronization isactive may be set, for example.

In addition, a management user for the operation object related to thereproduction may be set. For example, a display object indicating themanagement user may be displayed along with the operation object for themanagement user.

According to the seventh modification example of the embodiment, theoperation object is reproduced on the basis of the aforementionedpredetermined operation as described above. Therefore, a plurality ofpersons can control the same device to be operated. Therefore, it is notnecessary to displace the operation object to a user who desires tooperate the operation object, and it is possible to improve usability.

In addition, synchronization between the reproduced operation object andthe operation object as the reproduction source is controlled on thebasis of the form of the operation subject of the aforementionedpredetermined operation. Therefore, the user can select thesynchronization of the operation object in accordance with situations.For example, it is desirable to share information such as which programis being selected by all users in a case of an operation object such asa television program table, and it is thus considered that the operationobject is to be synchronized. Meanwhile, since it is considered that theindividual users perform different operations to an operation objectsuch as a controller for a video game, it is considered that theoperation object cannot be synchronized. In this manner, it is possibleto improve usability by enabling selection of the synchronization inaccordance with situations.

6. Hardware Configuration According to One Embodiment of the PresentDisclosure

The information processing apparatus 100 according to each embodiment ofthe present disclosure has been described above. The aforementionedprocessing of the information processing apparatus 100 is realized bycooperation of software and hardware of the information processingapparatus 100, which will be described below.

FIG. 61 is an explanatory diagram illustrating a hardware configurationof the information processing apparatus 100 according to an embodimentof the present disclosure. As illustrated in FIG. 61, the informationprocessing apparatus 100 includes a processor 131, a memory 132, abridge 133, a bus 134, an interface 135, an input device 136, an outputdevice 137, a storage device 138, a drive 139, a connection port 140,and a communication device 141.

(Processor)

The processor 131 functions as an arithmetic processing device andrealizes the functions of the recognition unit 101, the device selectionunit 102, the projection control unit 103, and the device control unit104 in the information processing apparatus 100 in cooperation withvarious programs. The processor 131 causes a variety of logicalfunctions of the information processing apparatus 100 to operate byexecuting the programs stored in the memory 132 or another storagemedium by using a control circuit. For example, the processor 131 can bea central processing unit (CPU), a graphics processing unit (GPU), adigital signal processor (DSP), or a system-on-a-chip (SoC).

(Memory)

The memory 132 stores programs, arithmetic parameters, or the like thatare used by the processor 131. For example, the memory 132 includes arandom access memory (RAM) and temporarily stores programs that are usedby being executed by the processor 131 or parameters and the like thatare appropriately changed in the execution. In addition, the memory 132includes a read only memory (ROM), and the RAM and the ROM realize thefunctions of the storage unit. Note that an external storage device maybe utilizes as a part of the memory 132 via the connection port 140, thecommunication device 141, or the like.

Note that the processor 131 and the memory 132 are connected to eachother by an internal bus that includes a CPU bus or the like.

(Bridge and Bus)

The bridge 133 establishes connection between the buses. Specifically,the bridge 133 connects the internal bus to which the processor 131 andthe memory 132 are connected and the bus 134 to which the interface 135is connected.

(Input Device)

The input device 136 is used by the user to operate the informationprocessing apparatus 100 or to input information to the informationprocessing apparatus 100. For example, the input device 136 includes aninput means for the user inputting information, an input control circuitthat generates an input signal on the basis of the input by the user andoutputs the input signal to the processor 131, and the like. Note thatthe input means may be a mouse, a keyboard, a touch panel, a switch, alever, a microphone, or the like. The user of the information processingapparatus 100 can input various kinds of data or provide instructionsfor processing operations to the information processing apparatus 100 byoperating the input device 136.

(Output Device)

The output device 137 is used to notify the user of information andrealizes the functions of the input and output unit. The output device137 may be a display device or a sound output device. For example, theoutput device 137 may be an apparatus such as a liquid crystal display(LCD) device, an organic light emitting diode (OLED) device, aprojector, a speaker, or a headphone or a module that provides outputsto such an apparatus.

Note that the input device 136 or the output device 137 may include aninput and output device. For example, the input and output device may bea touch screen.

(Storage Device)

The storage device 138 is a device for storing data. The storage device138 may include a storage medium, a recording device that records datain a recording medium, a reading device that reads data from a recordingmedium, a deleting device that deletes data recorded in a recordingmedium, and the like. The storage device 138 stores programs that theCPU 131 executes and various kinds of data.

(Drive)

The drive 139 is a reader/writer for a recording medium and isincorporated in or externally attached to the information processingapparatus 100. The drive 139 reads information stored in a mountedremovable storage medium such as a magnetic disk, an optical disc, amagneto-optical disc, or a semiconductor memory and outputs theinformation to the memory 132. In addition, the drive 139 can also writeinformation in the removable recording medium.

(Connection Port)

The connection port 140 is a port for connecting equipment directly tothe information processing apparatus 100. For example, the connectionport 140 can be a universal serial bus (USB) port, an IEEE1394 port, asmall computer system interface (SCSI) port, or the like. Alternatively,the connection port 140 may be an RS-232C port, an optical audioterminal, a High-Definition Multimedia Interface (HDMI) (registeredtrademark) port, or the like. Data may be exchanged between theinformation processing apparatus 100 and the external equipment byconnecting the external equipment to the connection port 140.

(Communication Device)

The communication device 141 intermediates communication between theinformation processing apparatus 100 and an external device and realizesthe functions of the communication unit 105. Specifically, thecommunication device 141 executes communication in accordance with awireless communication scheme or a wired communication scheme. Forexample, the communication device 141 executes wireless communication inaccordance with a cellular communication scheme such as Wideband CodeDivision Multiple Access (WCDMA) (registered trademark), WiMAX(registered trademark), long term evaluation (LTE), or LTE-A. Note thatthe communication device 141 may execute wireless communication inaccordance with an arbitrary wireless communication scheme such as anear-field wireless communication scheme such as Bluetooth (registeredtrademark), near field communication (NFC), a wireless USB, or aTransferJet (registered trademark) or a wireless local area network(LAN) such as Wi-Fi (registered trademark). In addition, thecommunication device 141 may execute wired communication such as signalline communication or wired LAN communication.

Note that the information processing apparatus 100 may not have a partof the configuration described above with reference to FIG. 61 or mayhave an arbitrary additional configuration. In addition, a one-chipinformation processing module on which an entirety or a part of theconfiguration described above with reference to FIG. 61 is integratedmay be provided.

7. Conclusion

As described above, according to the first embodiment of the presentdisclosure, candidates for a device to be operated, which is selectedthrough the first device selection based on a user's body form, arepresented to the user, and the user can select a device to be operatedfrom among the candidates. Therefore, it is not necessary for the userto move to the device to be operated first. In addition, it is possibleto suppress improper selection of the device to be operated and toprevent reselection of the device to be operated by the user selectingan operation target from among the presented candidates for the deviceto be operated. Also, it is possible to operate the device to beoperated without any specific device such as a remote controller and tosuppress occurrence of efforts to find the remote controller or the likeby the device to be operated being operated on the basis of selectionfrom among selection objects. Therefore, it is possible to reduce aburden on the user for selecting the device to be operated that the userdesires to operate.

In addition, according to the second embodiment of the presentdisclosure, it is possible to display the operation object withcomplication in accordance with user's conditions that are estimatedfrom the user's body form. Therefore, it is possible to enhancepossibilities that operation objects suitable for an operation that theuser desires are respectively displayed in each user's condition.Accordingly, it is possible to suppress variations in accordance withuser's conditions in relation to user's satisfaction about the operationobjects.

In addition, according to the third embodiment of the presentdisclosure, it is possible to control the display location of theoperation object not only in accordance with a user's direct operationon the operation object but also in accordance with an indirectoperation, conditions of the reference of the display locationirrelevant to user's actions, and the like by the reference of thedisplay location of the operation object being controlled. Therefore,the user can arrange or displace the operation object as if the userhandled a real object. Accordingly, it is possible to operate thedisplacement location of the operation object with a sense of moving anactual object.

The preferred embodiment(s) of the present disclosure has/have beendescribed above with reference to the accompanying drawings, whilst thepresent disclosure is not limited to the above examples. A personskilled in the art may find various alterations and modifications withinthe scope of the appended claims, and it should be understood that theywill naturally come under the technical scope of the present disclosure.

Although the selection objects and the operation objects are projectedin the aforementioned embodiments, for example, the present technologyis not limited to such an example. Specifically, the selection objectsand the operation objects may be visually recognized by the user bybeing superimposed on an external world image. For example, an imagerelated to the selection objects are superimposed on the external worldimage by the user wearing a display device (for example, an HUD (Head UpDisplay)) that transmits light of the external world image, anddisplaying an image related to the selection objects on a display unitof the display device, or projecting image light related to the imagefrom the display device to the user's eye. Note that the aforementioneddisplay device may be an HMD (Head Mount Display) on which the externalworld image and the image are displayed. In this case, the user canperceive the selection objects without the selection objects beingprojected in a real space. Therefore, it is possible to simplify theconfiguration of the information processing system 1 and to reduce costsand efforts for introduction of the information processing system 1. Inaddition, since the selection objects are not visually recognized byothers who do not wear the aforementioned display device, it is possibleto prevent eyesight of other persons to be disturbed. In addition, thesame is true for the operation objects.

Although the example in which devices such as a display device, an airconditioner device, a blower device, a recording device, an illuminationdevice, or a sound output device are controlled as devices to beoperated has been described in the aforementioned embodiments, otherdevices may be controlled. For example, the devices may be an electriccarpet, a microwave, a washing machine, a refrigerator, or a bathroomfacility.

Further, the effects described in this specification are merelyillustrative or exemplified effects, and are not limitative. That is,with or in the place of the above effects, the technology according tothe present disclosure may achieve other effects that are clear to thoseskilled in the art from the description of this specification.

Further, steps illustrated in the flowcharts of the above embodimentinclude not only processes which are chronologically performed inaccordance with the described order but also processes which are notnecessarily chronologically performed but performed in parallel orindividually as well. Further, it is needless to say that even in stepswhich are processed chronologically, the order can be appropriatelychanged depending on circumstances.

In addition, a computer program for causing a computer system to exhibitfunctions equivalent to those of the above-described respectivefunctional configurations of the information processing apparatus 100can be produced as well.

In addition, a storage medium in which the computer program is storedmay also be provided. Here, the computer system includes a singlecomputer like the hardware mounted in the information processingapparatus 100 or a plurality of computers that execute a series ofprocesses.

Additionally, the present technology may also be configured as below.

(1)

An information processing apparatus including:

-   -   a display control unit that controls display of selection        objects related to devices to be operated, which have been        selected through first device selection based on information        from which a form of a body of an operation subject directed to        the devices to be operated is estimated, and

a device control unit that controls the devices to be operated, whichhave been selected through second device selection based on informationrelated to a selecting operation performed by the operation subject tothe selection objects.

(2)

The information processing apparatus according to (1), in which theselection objects include objects indicating the devices to be operated,which have been selected through the first device selection.

(3)

The information processing apparatus according to (1) or (2), in whichthe selection objects are displayed such that the selection objects arevisually recognized in a form based on priority information.

(4)

The information processing apparatus according to (3), in which thepriority information includes information that is decided on the basisof information, from which the form of the body is estimated, in thefirst device selection.

(5)

The information processing apparatus according to (3) or (4), in whichthe priority information includes information that is decided on thebasis of biological information of the operation subject or informationrelated to a surrounding environment of the operation subject.

(6)

The information processing apparatus according to any one of (3) to (5),in which the priority information includes information that is decidedon the basis of information related to operations of the devices to beoperated in the past.

(7)

The information processing apparatus according to any one of (1) to (6),in which the display control unit controls display of operation objectsfor the devices to be operated, which have been selected through thesecond device selection.

(8)

The information processing apparatus according to any one of (1) to (7),in which the selection objects are displayed on a body of the operationsubject or in surroundings of the operation subject, the body or thesurroundings having a region in which the selection objects are able tobe displayed such that the selection objects are visually recognized bythe operation subject.

(9)

The information processing apparatus according to any one of (1) to (7),in which the selection objects are displayed at a location in accordancewith a decision operation of the first device selection performed by theoperation subject.

(10)

The information processing apparatus according to (9), in which thelocation in accordance with the decision operation includes a part ofthe body of the operation subject or the surroundings of the operationsubject designated through the decision operation.

(11)

The information processing apparatus according to (9) or (10), in whichthe location in accordance with the decision operation includes adisplay unit designated through the decision operation.

(12)

The information processing apparatus according to any one of (1) to(11), further including:

a notification control unit that controls a notification of the devicesto be operated, which have been selected through the first deviceselection when the selection objects are displayed.

(13)

The information processing apparatus according to (12), in which thenotification includes a display output that indicates a linkage betweenthe selected devices to be operated and the selection objects.

(14)

The information processing apparatus according to (12) or (13), in whichthe notification includes a sound output from a region in which theselected devices to be operated are situated.

(15)

The information processing apparatus according to any one of (1) to(14),

in which the form of the body includes a form of visual recognition ofthe operation subject, and

the devices to be operated, which are determined to fall within at leasta part of an eyesight of the operation subject, are selected through thefirst device selection.

(16)

The information processing apparatus according to any one of (1) to(15).

in which the form of the body includes a posture of the operationsubject, and

the devices to be operated, which are determined to fall within a regionthat is decided from the posture of the operation subject, are selectedthrough the first device selection.

(17)

The information processing apparatus according to any one of (1) to(16),

in which the form of the body includes motion of the operation subject,and

the devices to be operated, which are determined to fall within a regionthat is decided from the motion of the operation subject, are selectedthrough the first device selection.

(18)

The information processing apparatus according to any one of (1) to(17).

in which the form of the body includes speech generation from theoperation subject, and

the devices to be operated, which are determined to fall within a regionthat is decided from the speech generation from the operation subject,are selected through the first device selection.

(19)

An information processing method including, by using a processor:

controlling display of selection objects related to devices to beoperated, which have been selected through first device selection basedon information from which a form of a body of an operation subjectdirected to the devices to be operated is estimated; and

controlling the devices to be operated, which have been selected throughsecond device selection based on information related to a selectingoperation performed by the operation subject to the selection objects.

(20)

A program that is for causing a computer system to realize:

a display control function that controls display of selection objectsrelated to devices to be operated, which have been selected throughfirst device selection based on information from which a form of a bodyof an operation subject directed to the devices to be operated isestimated; and

a device control function that controls devices to be operated, whichhave been selected through second device selection based on informationrelated to a selecting operation performed by the operation subject tothe selection objects.

Additionally, the present technology may also be configured as below.

(21)

An information processing apparatus including:

an acquisition unit that obtains information related to a body form ofan operation subject; and

a display control unit that controls complication of an operation objectfor a device to be operated, which is visually recognized as if theoperation object was present in a real space, on the basis ofinformation related to the body form.

(22)

The information processing apparatus according to (21),

in which the body form includes a posture of the operation subject, and

the operation object is displayed such that the operation object isvisually recognized with complication in accordance with informationrelated to the posture of the operation subject.

(23)

The information processing apparatus according to (21) or (22),

in which information related to the body form includes biologicalinformation of the operation subject, and

the operation object is displayed such that the operation object isvisually recognized with complication in accordance with the biologicalinformation of the operation subject.

(24)

The information processing apparatus according to any one of (21) to(23).

in which the body form includes an action of the operation object, and

the operation object is displayed such that the operation object isvisually recognized with complication in accordance with informationrelated to the action of the operation subject.

(25)

The information processing apparatus according to any one of (21) to(24), in which the display control unit further controls complication ofthe operation object on the basis of information related to a locationat which the operation object is visually recognized, information fromwhich the operation subject is specified, or information from whichattributes of the operation subject are specified.

(26)

The information processing apparatus according to any one of (21) to(25), in which the operation object is displayed on a body of theoperation subject or in surroundings of the operation subject such thatthe operation object is visually recognized.

(27)

The information processing apparatus according to any one of (21) to(26), in which the operation object is displayed at a location inaccordance with a degree of information safety related to an operationof the device to be operated such that the operation object is visuallyrecognized.

(28)

The information processing apparatus according to any one of (21) to(27), in which the operation object is displayed at a location inaccordance with information from which the operation subject isspecified or information from which an attribute of the operationsubject is specified.

(29)

The information processing apparatus according to any one of (21) to(28), in which the operation object is displayed at a location inaccordance with the body form of the operation subject.

(30)

The information processing apparatus according to (29),

in which the body form of the operation subject includes a predeterminedmotion of the operation subject, and

the operation object is displayed at a location in accordance with thepredetermined motion.

(31)

The information processing apparatus according to (30), in which theoperation object includes the operation object associated with a targetof the predetermined motion.

(32)

The information processing apparatus according to (31), in which theoperation object associated with the target of the predetermined motionincludes the operation object for the device to be operated, which isthe target of the predetermined motion.

(33)

The information processing apparatus according to (31) or (32), in whichthe operation object associated with the target of the predeterminedmotion includes the operation object for the device to be operated,which is present in a real space same as a space of the target of thepredetermined motion.

(34)

The information processing apparatus according to (33), in which thedevice to be operated, which is present in the same real space as aspace of the target of the predetermined motion on the displayedoperation object is selected on the basis of information related to anenvironment in the same real space as the space of the target of thepredetermined motion, information related to a form of a person who ispresent in the same real space as the space of the target of thepredetermined motion, or time information.

(35)

The information processing apparatus according to any one of (21) to(34),

in which the operation object includes a notification operation objectfor a notification to the operation subject, and

the notification operation object is displayed in accordance withreception of the notification.

(36)

The information processing apparatus according to (35),

in which the notification to the operation subject includes anotification to a plurality of the operation subjects, and

the notification operation object is displayed at a location at whichthe notification operation object is visually recognized by each of theplurality of operation subjects.

(37)

The information processing apparatus according to (35) or (36),

in which the notification to the operation subject includes anotification to a specific operation subject, and

the notification operation object is displayed at a location at whichthe notification operation object is visually recognized only by thespecific operation subject.

(38)

The information processing apparatus according to (37),

in which the notification operation object is displayed in a case inwhich no persons other than the specific operation subject are presentin a space in which the specific operation subject is present.

(39)

An information processing method including, using a processor:

obtaining information related to a body form of an operation subject;and

controlling complication of an operation object for a device to beoperated, which is visually recognized as if the operation object waspresent in a real space, on the basis of information related to the bodyform.

(40)

A program for causing a computer system to achieve:

an acquisition function of acquiring information related to a body formof an operation subject; and

a display control function of controlling complication of an operationobject for a device to be operated, which is visually recognized as ifthe operation object was present in a real space, on the basis ofinformation related to the body form.

Additionally, the present technology may also be configured as below.

(41)

An information processing apparatus including:

a display control unit that controls display of an operation object fora device to be operated; and

a reference control unit that controls a reference of a location atwhich the operation object is displayed such that the operation objectis able to be visually recognized, on the basis of a predeterminedoperation performed to the operation object by an operation subject ofthe device to be operated.

(42)

The information processing apparatus according to (41),

in which the reference of the location includes an object in a realspace, and

the location at which the operation object is displayed is changed inaccordance with displacement of the object.

(43)

The information processing apparatus according to (42),

in which the object includes the operation subject, and

the operation object is displayed such that the operation object is ableto be visually recognized at a location in accordance with an attributeor a form of the operation subject.

(44)

The information processing apparatus according to any one of (41) to(43),

in which the reference of the location includes a position in a realspace, and

the operation object is displayed on the basis of the position in thereal space.

(45)

The information processing apparatus according to any one of (41) to(44), in which the display control unit further controls a type of thereference of the location in accordance with a form of the operationsubject when the predetermined operation is performed.

(46)

The information processing apparatus according to any one of (41) to(45), in which the display control unit controls a form of the operationobject when the reference of the location is controlled.

(47)

The information processing apparatus according to (46), in which theform of the operation object is controlled on the basis of informationrelated to an objective of an operation performed to the operationobject.

(48)

The information processing apparatus according to (46) or (47), in whichthe form of the operation object is controlled on the basis ofinformation related to the reference of the location before control orthe reference of the location after control.

(49)

The information processing apparatus according to (48), in which theinformation related to the reference of the location before control orthe reference of the location after control includes information fromwhich an attribute or a form of the reference of the location beforecontrol or of the reference of the location after control is specified.

(50)

The information processing apparatus according to any one of (46) to(49), in which the form of the operation object includes complication ofthe operation object or followability in response to displacement of thereference of the location.

(51)

The information processing apparatus according to any one of (41) to(50), in which the display control unit controls display of a displayobject from which a destination of changing of the reference of thelocation is specified.

(52)

The information processing apparatus according to any one of (41) to(51), in which in a case in which another operation object is present ata destination of changing of the reference of the location, the displaycontrol unit merges the operation objects whose the reference of thelocation is to be changed, and the another operation object at thedestination of changing of the reference of the location.

(53)

The information processing apparatus according to any one of (41) to(52), in which the reference control unit further controls an attributeof the operation object on the basis of the predetermined operation.

(54)

The information processing apparatus according to (53), in which theattribute of the operation object includes the operation subject foroperating the operation object.

(55)

The information processing apparatus according to (53) or (54), in whichthe display control unit controls a form of the operation object on thebasis of the attribute of the operation object.

(56)

The information processing apparatus according to any one of (41) to(55), in which the operation object is copied on the basis of thepredetermined operation.

(57)

The information processing apparatus according to (56), in whichsynchronization between a copy of the operation object and the operationobject that is a source of the copying is controlled on the basis of aform of the operation subject in the predetermined operation.

(58)

The information processing apparatus according to any one of (41) to(57),

in which the predetermined operation includes a set of a first operationand a second operation performed to the operation object, and

the reference control unit changes the reference of the operation objectselected through the first operation, on the basis of a location inaccordance with the second operation.

(59)

An information processing method including, using a processor:

controlling display of an operation object for a device to be operated;and

controlling a reference of a location at which the operation object isdisplayed such that the operation object is able to be visuallyrecognized, on the basis of a predetermined operation performed to theoperation object by an operation subject of the device to be operated.

(60)

A program for causing a computer system to achieve:

-   -   a display control function of controlling display of an        operation object for a device to be operated; and

a reference control function of controlling a reference of a location atwhich the operation object is displayed such that the operation objectis able to be visually recognized, on the basis of a predeterminedoperation performed to the operation object by an operation subject ofthe device to be operated.

REFERENCE SIGNS LIST

-   10 projection imaging device-   20 display device-   21 air conditioner device-   22 blower device-   100 information processing apparatus-   101 recognition unit-   102 device selection unit-   103 projection control unit-   104 device control unit-   105 communication unit-   106 storage unit

1. An information processing apparatus comprising: a display controlunit that controls display of an operation object for a device to beoperated; and a reference control unit that controls a reference of alocation at which the operation object is displayed such that theoperation object is able to be visually recognized, on a basis of apredetermined operation performed to the operation object by anoperation subject of the device to be operated.
 2. The informationprocessing apparatus according to claim 1, wherein the reference of thelocation includes an object in a real space, and the location at whichthe operation object is displayed is changed in accordance withdisplacement of the object.
 3. The information processing apparatusaccording to claim 2, wherein the object includes the operation subject,and the operation object is displayed such that the operation object isable to be visually recognized at a location in accordance with anattribute or a form of the operation subject.
 4. The informationprocessing apparatus according to claim 1, wherein the reference of thelocation includes a position in a real space, and the operation objectis displayed on a basis of the position in the real space.
 5. Theinformation processing apparatus according to claim 1, wherein thedisplay control unit further controls a type of the reference of thelocation in accordance with a form of the operation subject when thepredetermined operation is performed.
 6. The information processingapparatus according to claim 1, wherein the display control unitcontrols a form of the operation object when the reference of thelocation is controlled.
 7. The information processing apparatusaccording to claim 6, wherein the form of the operation object iscontrolled on a basis of information related to an objective of anoperation performed to the operation object.
 8. The informationprocessing apparatus according to claim 6, wherein the form of theoperation object is controlled on a basis of information related to thereference of the location before control or the reference of thelocation after control.
 9. The information processing apparatusaccording to claim 8, wherein the information related to the referenceof the location before control or the reference of the location aftercontrol includes information from which an attribute or a form of thereference of the location before control or of the reference of thelocation after control is specified.
 10. The information processingapparatus according to claim 6, wherein the form of the operation objectincludes complication of the operation object or followability inresponse to displacement of the reference of the location.
 11. Theinformation processing apparatus according to claim 1, wherein thedisplay control unit controls display of a display object from which adestination of changing of the reference of the location is specified.12. The information processing apparatus according to claim 1, whereinin a case in which another operation object is present at a destinationof changing of the reference of the location, the display control unitmerges the operation objects whose the reference of the location is tobe changed, and the another operation object at the destination ofchanging of the reference of the location.
 13. The informationprocessing apparatus according to claim 1, wherein the reference controlunit further controls an attribute of the operation object on a basis ofthe predetermined operation.
 14. The information processing apparatusaccording to claim 13, wherein the attribute of the operation objectincludes the operation subject for operating the operation object. 15.The information processing apparatus according to claim 13, wherein thedisplay control unit controls a form of the operation object on a basisof the attribute of the operation object.
 16. The information processingapparatus according to claim 1, wherein the operation object is copiedon a basis of the predetermined operation.
 17. The informationprocessing apparatus according to claim 16, wherein synchronizationbetween a copy of the operation object and the operation object that isa source of the copying is controlled on a basis of a form of theoperation subject in the predetermined operation.
 18. The informationprocessing apparatus according to claim 1, wherein the predeterminedoperation includes a set of a first operation and a second operationperformed to the operation object, and the reference control unitchanges the reference of the operation object selected through the firstoperation, on a basis of a location in accordance with the secondoperation.
 19. An information processing method comprising, using aprocessor: controlling display of an operation object for a device to beoperated; and controlling a reference of a location at which theoperation object is displayed such that the operation object is able tobe visually recognized, on a basis of a predetermined operationperformed to the operation object by an operation subject of the deviceto be operated.
 20. A program for causing a computer system to achieve:a display control function of controlling display of an operation objectfor a device to be operated; and a reference control function ofcontrolling a reference of a location at which the operation object isdisplayed such that the operation object is able to be visuallyrecognized, on a basis of a predetermined operation performed to theoperation object by an operation subject of the device to be operated.